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Overview 



In low-performing public high schools in U.S. cities, high proportions of students drop 
out, students who stay in school typically do not succeed academically, and efforts to make sub- 
stantial reforms often meet with little success. The Talent Development High School model is a 
comprehensive school reform initiative that has been developed to address these challenges. Tar- 
geting some of the most troubled schools in the country, the model seeks to raise the expectations 
of teachers and students and to prepare all students for postsecondary education and employment. 

MDRC, a nonpartisan, nonprofit education and social policy research organization, con- 
ducted an independent, third-party evaluation of Talent Development. This rigorous evaluation 
focuses on the first five high schools to begin using the model in the School District of Philadel- 
phia. The evaluation follows 20 cohorts of ninth-grade students for up to four years of high 
school using a comparative interrupted time series research design. 

Key Findings 

• Talent Development produced substantial gains in attendance, academic 
course credits earned, and promotion rates during students’ first year of high 
school. These impacts emerged in the first year of implementation and were repro- 
duced as the model was extended to other schools in the district and as subsequent 
cohorts of students entered the ninth grade. 

• Talent Development’s strong positive impacts during the first year of high 
school are consistent with the model’s intensive initial focus on the ninth 
grade and its emphasis on combining high-quality curricular and instructional en- 
hancements with pervasive structural reforms aimed at building supportive and 
personalized learning environments. 

• The improvements in credits earned and promotion rates for ninth-graders 
were sustained as students moved through high school. Improvements in stu- 
dent performance on the eleventh-grade state standards assessment began to 
emerge for later cohorts of students as the most intensive components of the model 
were extended beyond the ninth grade. There are also early indications that Talent 
Development is improving graduation rates. 

The findings in this report provide encouraging evidence that real improvements can be 
made in some of the lowest-performing high schools in the country. Even with Talent Develop- 
ment’s substantial and persistent positive impacts, however, large proportions of the students in 
these high schools are not making adequate progress toward graduation. Moreover, because this 
evaluation focuses on Talent Development’s initial scaling-up effort in a single school district, it 
is not clear what would be required to produce the same effects in a larger number of schools or 
in other settings. 
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Preface 



The state of urban high schools in the United States — long a cause for concern — is 
now recognized as a national crisis. In too many city high schools in high-poverty neighbor- 
hoods, less than half of entering ninth-graders eventually graduate — and the students who do 
receive a diploma are often unprepared for work or postsecondary education. President Bush 
and state governors, among other leaders, are calling for change. 

The Talent Development High School model is a reform initiative now being used in 
more than 80 schools in 20 districts nationwide. This report shows that the model made a posi- 
tive difference in Philadelphia, where it was first brought to scale. Using an unusually rigorous 
research method, MDRC found that Talent Development improved first-time ninth-graders’ 
attendance, accrual of academic credits (particularly in algebra), and promotion rates into tenth 
grade. Results from the first two schools to begin using the model suggest that Talent Develop- 
ment had a positive effect on graduation rates as well. 

The best-implemented component of the intervention in Philadelphia seemed to be the 
Ninth Grade Success Academy, which transformed the ninth-grade experience in these schools. 
Ninth-graders were grouped together in small “learning communities” that were taught exclu- 
sively by a handful of dedicated teachers. They received instruction with specialized curricula in 
extended classes — with double doses of English and math. 

The Talent Development schools benefited from extra funding and extensive technical 
assistance from the Center for Research on the Education of Students Placed At Risk 
(CRESPAR), the developer of the program model. However, it is important to note that these 
impressive results occurred without much institutional support from the school district itself. 

While these findings are encouraging, they also highlight how far we still have to go. 
Even in the most successful Talent Development schools, large proportions of entering ninth- 
graders still do not graduate in four years. 

Still, these findings provide some of the most reliable evidence available that change is 
possible in the nation’ s most troubled high schools. The challenge is to resist the temptation to 
look for silver bullet solutions — building instead on what we now know about what works. 

Gordon L. Berlin 
President 
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Executive Summary 



Low-performing public high schools in U.S. cities are often seen as places of little hope. 
High proportions of students drop out, many students who stay in school are not well prepared for 
college or the workforce, and efforts to make substantial improvements in ailing schools tend to 
meet with little success. Two recent trends offer some good news, however. Educators and poli- 
cymakers — including President Bush, state governors, and foundation and business leaders — 
have recently recommitted themselves to addressing the challenge of reforming secondary educa- 
tion in urban settings. In addition, a number of comprehensive school reform models have been 
developed over the past ten years, and some have begun to show evidence of being effective. 

This report offers encouraging findings on one such initiative: the Talent Development 
High School model. Talent Development, which targets some of the most troubled schools in 
the country, seeks to raise the expectations of teachers and students, with the ultimate goal of 
preparing all students for postsecondary education and employment. The evaluation took place 
in the School District of Philadelphia, using a particularly rigorous research design that provides a 
high level of confidence about Talent Development’s effectiveness — essentially allowing re- 
searchers to conclude that changes in student engagement and performance are, indeed, due to 
Talent Development. In summary, the key findings of this final report from the study are: 

• Talent Development produced substantial gains in attendance, academic 
course credits earned, and promotion rates during students’ first year of 
high school. These impacts emerged in the first year of implementation and 
were reproduced as the model was extended to other schools in the district 
and as subsequent cohorts of students entered the ninth grade. 

• Talent Development’s strong positive impacts during the first year of 
high school are consistent with the model’s intensive initial focus on the 
ninth grade and its emphasis on combining high-quality curricular and in- 
structional enhancements (including offering transitional math and English 
courses, creating teaching teams, and providing ongoing coaching for teach- 
ers) with pervasive structural reforms (including developing small learning 
communities and using extended class periods) aimed at building supportive 
and personalized learning environments. 

• The improvements in credits earned and promotion rates for ninth - 
graders were sustained as students moved through high school. Im- 
provements in student performance on the eleventh-grade state standards as- 
sessment began to emerge for later cohorts. There are also early indications 
that Talent Development is improving graduation rates. 
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Even with Talent Development's substantial and persistent positive impacts, the schools 
still have a long way to go to achieve the initiative’ s vision of preparing all students for gradua- 
tion, postsecondary education, and employment. For instance, even in the Talent Development 
schools in Philadelphia, more than half of first-time ninth-grade students will not be ready to 
graduate in four years. Also, because this evaluation focuses on Talent Development’s initial 
scaling-up effort in the School District of Philadelphia, it is not clear what would be required to 
produce the same effects in a larger number of schools and in more diverse contexts. 



How Was Talent Development Implemented in Philadelphia? 

Talent Development was initiated in 1994 through a partnership between the Center for 
Research on the Education of Students Placed At Risk (CRESPAR) and Patterson High School 
in Baltimore, Maryland. In 1998, CRESPAR, in collaboration with the Philadelphia Education 
Fund, a local educational intermediary organization, began Talent Development’s first and 
most ambitious scaling-up effort in Philadelphia. 

Throughout the 1990s, nonselective high schools in Philadelphia faced a growing num- 
ber of challenges. More than three-quarters of students in the district entered the ninth grade 
with reading and math skills below grade level, and over half could be considered chronic ab- 
sentees (that is, students who miss an average of one of every five school days each year). 
Moreover, each year, fewer than two-thirds of ninth-graders were promoted to the tenth grade. 
For those who continued on to the upper grades, only about 10 percent performed at or above 
grade level on standardized state tests, and less than 40 percent were on schedule to graduate 
four years after starting high school. 

As of the 2003-2004 school year, seven of the district’s 22 nonselective high schools 
were implementing the model. This report focuses on the five schools that began using the 
model first. Following are the key features of the scaling-up process in these five schools: 

• The Ninth Grade Success Academy was the most strongly and consis- 
tently implemented element of the Talent Development model. 

The schools and CRESPAR made immediate changes to both the structure and the instruc- 
tional core of the entire ninth grade in each high school. This included relocating all ninth-grade 
classes to a single floor or wing and creating “learning communities” — small, self-contained aca- 
demic teams of 100 to 125 students taught exclusively by the same four or five teachers. Each 
school modified its daily schedule to include blocks of four 80- to 90-minute classes and changed 
the sequence of courses to allow students to take “double doses” of math and English over the 
course of the school year (that is, to take CRESPAR' s first-term “transition” courses in math and 
strategic reading as electives in order to prepare for courses required in the second term, including 
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algebra and English language arts). Each school also provided students with a “Freshman Semi- 
nar,” developed by CRESPAR and designed to help students develop solid study skills and to fo- 
cus on personal and social adjustment issues. 

• The schools made more limited progress in transforming the upper 
grades into Career Academy programs that were fully aligned with the 
principles and structures of the Talent Development model. 

During much of the period covered by this evaluation, CRESPAR and its partners 
worked to refine the model for students in grades ten through twelve — centered around Career 
Academies, which combine the structure of small learning communities with curricular choices 
built around broad career themes. While many of the high schools already had existing Career 
Academies, many did not have cohesive small learning communities, and the curriculum com- 
ponents were of uneven quality. Specialized courses in math and English, team-teaching, extra 
supports for struggling students, and teacher professional development were phased in and 
strengthened for the upper grades during the later stages of the follow-up period for this study. 

• Implementation teams at each school, professional development oppor- 
tunities, and the expertise of CRESPAR’s staff were important facets of 
the implementation process. 

In each school, a small team consisting of a part-time coordinator and part-time curricu- 
lar coaches helped establish and maintain the model. These teams were supported by a full-time 
coordinator for the district and by curriculum developers and school reform experts at 
CRESPAR. 



• Implementation of Talent Development was sanctioned by the district, but 
it received neither formal endorsement nor direct institutional support 

The lack of formal district endorsement, along with variation in the context and in the 
operational support provided at individual schools, led to variation in the quality and depth of 
implementation of Talent Development across schools and over time. Nevertheless, each of the 
five schools that are the focus of this report was able to implement the model with a reasonable 
level of fidelity and intensity. 



How Was the Impact of Talent Development Evaluated? 

MDRC used a research method called a “comparative interrupted time series analysis” to 
estimate the effect of Talent Development on student outcomes. The first step in estimating im- 
pacts with this design is to measure the change at Talent Development schools in a given student 
outcome, after the school began using the model, relative to the average outcome during a pre- 
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implementation baseline period. This estimation represents how student performance changed in 
the presence of Talent Development. The next step is to measure the corresponding change during 
the same period for similar schools not implementing the model. This measurement provides an 
estimate of how student performance would have changed at the Talent Development schools in 
the absence of the reform. The difference between these two changes is an estimate of the impact 
of the intervention — what Talent Development caused to happen. 

The design for this evaluation is particularly rigorous, addressing many concerns typi- 
cally raised about research that does not use random assignment. While no quasi-experimental 
methodology irrefutably establishes causality, this version of the comparative interrupted time 
series method provides a strong basis on which to attribute changes in student performance to 
Talent Development. 



Did Talent Development Make a Difference? 

The report focuses on program impacts for three cohorts of first-time ninth-graders from 
each of five Talent Development high schools. It includes findings from these students’ first year 
of high school and follows at least one cohort from each school for a full three years to the point at 
which they should be in eleventh grade and taking the state’s standardized test in math and read- 
ing. The study was also able to follow ninth-graders from two of the schools through four years to 
examine Talent Development’s impact on graduation rates. The findings presented in the report 
support the following conclusions: 

• For first-time ninth-grade students, Talent Development produced sub- 
stantial gains in attendance, academic course credits earned, and pro- 
motion rates during the students’ first year of high school. 

Figure ES-1 provides a summary of the key impact findings. The solid bars represent 
the changes in key outcomes for the Talent Development schools between the baseline period 
and the follow-up period. The white bars represent changes in key outcomes for the comparison 
schools during the same period. The difference between the two bars represents the impact of 
Talent Development. The numbers below the bars represent the baseline averages for key out- 
comes (such as attendance rate and basic academic curriculum completed) for first-time ninth- 
grade students in Talent Development high schools — that is, the performance levels that Talent 
Development had to build upon. 
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The Talent Development Evaluation 
Figure ES.l 

Impacts on and Baseline Averages for Key Outcomes 
for First-Time Ninth-Grade Students 



40 




-5 J Attendance Basic academic 

rate curriculum 

completed 



Baseline averages 
for Talent 
Development 
schools (%) 



72.8 



43.5 



Impact = 24.5*** 



28.0 



3.5 



Algebra 
credit earned 



33.1 



■ Talent Development Schools 
□ Non-Talent Development Schools 



Impact = 8.0*** 




Promoted to 
the 10th 
grade 

59.0 



Impact = 6.5* 




-0.45 

Promoted to 



the 11th 
grade 

47.2 



SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent Development high 
schools. Students in the sample were included on the district's transcript and attendance records. The sample excludes students 
who did not attempt at least one credit during a given school year. First-time 9th-grade students were defined as students whose 
records indicate that they were in the 9th grade in the year under study and in the 8th grade in the previous year's administrative 
data file. 

Baseline averages for Talent Development schools show the average outcome levels for students in these schools during the 
three-year period prior to the implementation of Talent Development. Each bar in the graph represents the change from baseline 
averages in Talent Development and non-Talent Development schools. The estimated impact of Talent Development is the 
difference in deviations from the baseline average between the Talent Development and non-Talent Development schools. A 
two-tailed t-test was applied to the impacts at follow-up. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 
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The first set of bars in the figure shows that attendance rates in the Talent Development 
schools improved by an average of about 5 percentage points, while the rates remained constant 
in the comparison schools. Talent Development’s estimated impact on the attendance rates of 
first-time ninth-graders was 5 percentage points — or an average increase of about nine extra 
school days per year for each student in a Talent Development high school. 

Beyond encouraging students to attend school more regularly, a major goal for Talent 
Development is to help ninth-graders complete a more rigorous complement of courses dur- 
ing their first year of high school. Figure ES-1 shows that the Talent Development schools 
consistently outpaced their non-Talent Development counterparts in increasing the percentage 
of students who completed what could be considered a basic academic curriculum: earning at 
least five credits during the school year, with three of those credits being in math, English, and 
science. On average, Talent Development increased the percentage of students completing a 
basic academic curriculum by about 8 percentage points over what was achieved in the non- 
Talent Development schools. Although not shown in the figure, Talent Development also in- 
creased total credits earned during the ninth grade by about two-thirds of a full-year credit. 

Most notably, Talent Development produced a substantial increase in the proportion of 
students who earned a credit in algebra — nearly 25 percentage points. Algebra is a critical 
“gate-keeping” course, one that is usually required of students both for high school graduation 
and for admission to college. For a typical class of about 500 first-time ninth-graders, the esti- 
mated impact of nearly 25 percentage points means that Talent Development adds nearly 125 
students to the rolls of those who earn a credit in algebra each year. 

Finally, Talent Development also improved the rates at which students were promoted 
from the ninth grade to the tenth (including both end-of-year and mid-year promotions). Figure 
ES-1 shows that promotion rates in Talent Development schools rose by nearly 10 percentage 
points after the program began implementation. During the same time period, these rates rose 
by only about 2 percent in the comparison schools. The resulting impact of 8 percentage points 
on promotion to the tenth grade means that an additional 40 ninth-graders per school, per year, 
made the transition from the ninth to the tenth grade — which is generally considered the weak- 
est point in the educational pipeline. 

• The impacts on credits earned and on promotion rates were sustained as 
first-time ninth-graders moved through high school. 

Figure ES-1 also shows that Talent Development improved the rate at which students 
were promoted to the eleventh grade by about 6 percentage points. Though not shown in the 
figure, Talent Development also sustained its impact on the accumulation of key academic 
course credits through the eleventh grade. The model produced a 10 percentage point increase 
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in the percentage of students who had earned a minimum of three math and three English cred- 
its — a key threshold for staying on course for graduation. 



• Talent Development produced marginal improvements in math test 
scores among early cohorts of eleventh-graders, but stronger improve- 
ments occurred for later cohorts of students who were exposed to a 
more intensive version of the model. 

For the first cohorts of eleventh-grade students in the Talent Development high schools, 
the model produced a modest increase of 6 percentage points in the percentage of students who 
scored at the basic level or above on the math portion of the state’s standardized assessment. For 
later cohorts of eleventh-grade students in two of the Talent Development high schools, the 
model substantially increased the average scaled scores in reading and math — by effect sizes 
of 0.38 and 0.65, respectively. 

• Based on evidence from only the first two schools to implement the 
model, Talent Development appears to have produced positive impacts 
on high school graduation rates. 

Because they had been working with Talent Development for a full five years, the first 
two high schools to implement the model provide the opportunity to examine impacts on high 
school graduation rates. For the first two cohorts of first-time ninth-graders in these high 
schools, Talent Development improved the likelihood of graduating on time by about 8 percent- 
age points. In other words, for a typical entering ninth-grade class of 500 students, Talent De- 
velopment was able to produce an average of about 40 new graduates per year. 

• Although the likelihood of repeating the ninth grade declined in Talent 
Development high schools (due to the model’s impact on promotion 
rates), the results for those students who did need to repeat the ninth 
grade were mixed. 

Prior to Talent Development, about one-third of the ninth-grade students in the study 
schools were repeating the grade for at least the first time. As noted above, Talent Development 
did increase the rate at which first-time ninth-graders were promoted to the tenth grade. Among 
those who still needed to repeat the grade, Talent Development produced an increase of more 
than 5 percentage points in average attendance rates. Also, the Talent Development high 
schools saw notable increases in the credits earned by repeating ninth-graders — yet these 
trends were present in the non-Talent Development schools as well. Overall, however, Talent 
Development actually increased the likelihood that students who repeated a full year of ninth 
grade would leave the school system before the end of their fourth year of high school. 
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Taking Stock of the Results 

Talent Development produced substantial and pervasive improvements in outcomes for 
first-time ninth-grade students in very low-performing high schools. In a high school of 500 
first-time ninth-graders, Talent Development adds about nine days of school attendance for each 
student and helps an extra 125 students pass algebra, an extra 40 students achieve promotion to 
the tenth grade, and an extra 40 students graduate on time. 

Nevertheless, two important cautions are worth noting. First, the schools that have been 
using the model still have a great deal left to accomplish if they are to reach the model’s aspira- 
tion of preparing all students for graduation, postsecondary education, and employment. Even in 
a Talent Development school, a typical ninth-grader will still miss about 40 days of school, 
nearly a third will not be promoted to the tenth grade, and more than half will not be ready to 
graduate within four years. Thus, even successful interventions like Talent Development still 
need much more power. Toward that end, the Talent Development model continues to evolve 
by strengthening the upper-grade components, particularly by extending curricular and instruc- 
tional reforms to tenth and eleventh grades. 

Second, achieving these initial positive results required significant extra funds and very 
demanding changes to school organization, instruction, and teacher support. The estimated ad- 
ditional cost of operating Talent Development is approximately $250 to $350 per student, per 
year. In Philadelphia, the costs were covered by federal grants, in-kind and direct contributions 
from the school district, and other funds available to CRESPAR. Despite the financial support 
for Talent Development from the district, however, these impressive results were accomplished 
without the district’s formal endorsement and without deeper institutional support. The effec- 
tiveness of Talent Development and other comprehensive school reforms are likely to be en- 
hanced and sustained when school districts focus staffing and leadership decisions on specific 
school improvement strategies and marshal the funding and resources to support them. 

In conclusion, the findings in this report provide encouraging evidence that real im- 
provements can be made in some of the lowest-performing high schools in the country — if 
there exists a sustained investment in developing the skills to deal with poorly prepared students 
and weak learning environments, and if that investment is built on reasonable fidelity to the ten- 
ets and components of a well-conceived reform approach. 
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Introduction 

This report provides an independent assessment of the impact of the Talent Develop- 
ment High School model on student performance in five high schools in the School District of 
Philadelphia. 1 The Talent Development High School model was initiated in 1994 through a 
partnership between the Center for Research on the Education of Students Placed At Risk 
(CRESPAR), based at The Johns Hopkins University and Howard University, and Patterson 
High School in Baltimore, Maryland. In 1998, CRESPAR, in collaboration with the Philadel- 
phia Education Fund, began Talent Development’s first and most ambitious scaling-up effort in 
Philadelphia. As of the 2003-2004 school year, seven of the district’s 22 nonselective high 
schools were implementing the model. 2 Nationwide, Talent Development is operating in 83 
high schools in 32 school districts in 20 states, and CRESPAR is seeking ways to refine the 
model and expand it further. 

The Talent Development model targets schools that face serious problems with student 
attendance, discipline, achievement scores, and dropout rates to help transform them structur- 
ally, instructionally, and comprehensively (across all grades and departments). The organiza- 
tional and curriculum-related changes that constitute Talent Development aim to establish a 
strong, positive school climate for learning, promote high standards for English and mathemat- 
ics coursework for all students, and provide professional development systems to support im- 
plementation of the recommended reforms. Each of these objectives is part of a larger set of 
goals: to enhance students’ school attendance, improve student learning, and keep students on 
course toward high school graduation. 

MDRC is conducting the independent, third-party evaluation of the model, funded by 
the U.S. Department of Education through its Institute of Education Sciences, as part of the 
Comprehensive School Reform Demonstration program. 2 In 1999, CRESPAR received a grant 
from the Institute of Education Sciences (then called the Office of Educational Research and 
Improvement [OERI]) to “scale up” the Talent Development model in several school districts 



'The impact analysis in this report focuses on the first five high schools to adopt Talent Development in 
Philadelphia. The discussion of the implementation of Talent Development in Philadelphia discusses these five 
schools and another two high schools that adopted the model in the 2003-2004 school year. An eighth school 
adopted Talent Development in the 2002-2003 school year, but it has not been included in either analysis be- 
cause it began the process of closing the following year. 

2 Nonselective schools typically enroll students from a nearby neighborhood and do not require them to 
meet academic or other performance standards for admission. To protect the anonymity of the schools high- 
lighted in this report, letters are used to refer to specific schools. 

3 For more information on the Comprehensive School Reform Demonstration program, see 
http://www.ed.gov/offices/OERPcsrrdp.html. 
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around the country. A requirement of the OERI funding was that CRESPAR had to contract 
with an external evaluator to conduct an independent assessment of Talent Development’s im- 
pact on student performance. Through a competitive proposal process, CRESPAR chose 
MDRC to conduct the evaluation. This is the final report from that evaluation. 4 

This report is being released as educators and policymakers (led by President George 
W. Bush), the state governors, and foundation and business leaders have turned the nation’s at- 
tention to the problems facing low-performing high schools. So far, much of the attention has 
focused on framing the problem as a national “crisis,” ratcheting up academic standards, and 
establishing assessment and accountability systems to monitor student performance and hold 
students and teachers accountable for results. To this point, little attention has as yet been paid 
to the challenges of equipping low-performing high schools with the tools needed to prevent 
students from dropping out, ensure that they have the support and basic skills they will need to 
complete high-level academic work, and teach them what they need to know to succeed on the 
high-stakes assessments. Fortunately, evidence about the effectiveness of particular approaches 
to these challenges is beginning to emerge. The Talent Development evaluation is one source of 
such evidence. 

A preliminary report from MDRC’s evaluation of the Talent Development High School 
model was released in June 2004. It focused primarily on Talent Development’s effects on 
ninth-graders during the first three years of the model’s implementation. The findings showed 
that, for first-time ninth-grade students, Talent Development produced substantial gains in aca- 
demic course credits and promotion rates and modest improvements in attendance. 

The current report provides a more detailed account of how Talent Development was 
implemented in Philadelphia and includes two additional years of follow-up data on student 
performance. The available data follows ninth-grade students through up to four years of high 
school, and the analysis examines the impact of Talent Development on attendance, course tak- 
ing, and promotion outcomes. For the two high schools that began working with Talent Devel- 
opment first, the report provides some initial insights into the effect Talent Development may 
be having on the rate at which students are able to graduate on time. 

The report is organized into six sections: 



4 See Kemple and Herlihy (2004) for the preliminary report on the Talent Development High School 
model. A Talent Development Middle School model also exists and also has been the subject of an MDRC 
evaluation. See Herlihy and Kemple (2004) for information about the effects of the Talent Development Mid- 
dle School model. 
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• The first section provides evidence of the problems faced by the high schools 
in Philadelphia, where the Talent Development High School model not only 
began its scaling-up process but also has been used most pervasively. 

• The second section describes the model itself, focusing on the features and 
components that aim to help high schools attack the problems they face as di- 
rectly and immediately as possible. 

• The third section describes the implementation of Talent Development in 
Philadelphia, highlighting factors that appeared to have facilitated or hin- 
dered execution of the model. 

• The fourth section provides an overview of the analytic strategies being used 
to estimate the impact of Talent Development — by comparing changes in 
student outcomes in Talent Development schools with changes in student 
outcomes in schools that did not implement Talent Development. 

• The fifth section presents the findings from the analyses of Talent Develop- 
ment’s impact on student performance in the five high schools that began using 
the model first. It examines the model’s effect on ninth-graders’ attendance 
rates, promotion rates, and credits earned toward graduation. This section ex- 
tends the impact analysis into students’ second, third, and fourth years of high 
school to assess the effect of Talent Development on course credits earned, 
promotion rates, test score performance, and on-time graduation rates. 

• The final section discusses lessons that may be drawn from the analyses and 
attempts to put these findings in the context of research from other compre- 
hensive school reform efforts. 

In the sections that present findings on implementation and impacts, the report provides 
further information about the data sources and methods used in the analyses. Finally, the report 
includes an appendix with additional tables from the impact analysis. 



The State of High Schools in Philadelphia 

Low-performing high schools in many U.S. cities reflect some of the deepest problems 
in American education. Recent data show that, in nearly 20 percent of the nation’s 1 1,000 regu- 
lar and vocational high schools, the number of enrolled students in an entering class of ninth- 
graders declines by nearly 40 percent during four years of high school. Such schools have been 
characterized as having “weak promoting power” and exhibit astonishing dropout rates and low 
performance even among their graduates. The number of these troubled schools increased by 60 
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percent between 1993 and 2002; the schools are concentrated in urban centers. In half of the 
nation’s 100 largest cities, the majority of regular and vocational high school students are en- 
rolled in schools where graduation is not the norm. 5 

Philadelphia provides a particularly rich and relevant context for a rigorous assessment of 
Talent Development. Its 22 nonselective high schools, and particularly the five that are the subject 
of this evaluation, are typical of the types of schools that Talent Development was specifically 
designed to help and in which Talent Development has been most widely implemented. Most of 
these high schools have been identified as having weak promoting power, 6 and together these 
schools exhibit the problems faced by many low-performing high schools across the country. 

Table 1 presents descriptive information for the 22 nonselective, comprehensive high 
schools in Philadelphia. The table captures the state of these schools over the three school years 
— 1996-1997 through 1998-1999 — before Talent Development began scaling up its model in 
several of the high schools. 7 During this period, the 22 high schools enrolled approximately 76 
percent of the high school students in the district. 8 Following is an overview of the context that 
Talent Development entered into when it began its scaling-up effort. 

Getting a Poor Start in the Ninth Grade 

Among the most troubling problems in large urban high schools is the degree to 
which high school students become disengaged from school and eventually drop out. While 
this process typically begins before students reach high school, its devastating effects are con- 
centrated in the ninth grade. The first column of Table 1 presents several indicators of the dif- 
ficulty that ninth-graders in the district’s nonselective high schools had in progressing further. 
For example, among ninth-grade students, 41 percent were already overage for their grade, 
indicating that they had already repeated a previous grade. In fact, at the time the data were 
collected, 27 percent of the ninth-graders were repeating the ninth grade. High levels of dis- 
engagement can be seen in the low attendance rates and high degree of chronic absenteeism 
(defined as having attendance rates of 80 percent or lower). The table shows that nearly 58 



5 See Balfanz and Legters, 2004. 

6 See Balfanz and Legters, 2004. 

7 Note that the sample of students that is the basis for the information in Table 1 excludes students who 
may have been listed in the district administrative records but who had not attempted at least one course credit. 
Such sUidents could not be included in the analysis because the district did not provide consistent information 
about them for the years included in this study. 

Approximately 19 percent of the district’s ninth- through twelfth-grade students attended selective high 
schools that admitted students on the basis of prior academic performance. Another 5 percent were enrolled in 
alternative schools or schools serving students with special needs. 
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The Talent Development Evaluation 

Table 1 

Characteristics of Students in Nonselective High Schools in 
the School District of Philadelphia, 

School Years 1996-1997 through 1998-1999 



Characteristic 


9th Grade 


10th- 12th Grade 


Overage for grade” (%) 


40.8 


35.4 


Currently repeating grade b (%) 


26.6 


11.4 


8th-grade SAT-9 test scores 
Reading Comprehension 
Average Normal Curve Equivalent (NCE) score 


36.6 


NA 


Percent scoring above grade level 


23.4 


NA 


Math total 


Average Normal Curve Equivalent (NCE) score 


35.3 


NA 


Percent scoring above grade level 


16.9 


NA 


llth-grade state assessment scores” 

Reading Comprehension 
Average Normal Curve Equivalent (NCE) score 


NA 


27.0 


Percent scoring above grade level 


NA 


10.2 


Math total 


Average Normal Curve Equivalent (NCE) score 


NA 


26.9 


Percent scoring above grade level 


NA 


8.6 


Attendance rate” (%) 


71.3 


77.1 


Students with an attendance rate of: (%) 


90% or higher 


23.3 


28.6 


80% or lower 


57.5 


47.3 


Earned 4 or more course credits for the year” (%) 


58.2 


76.6 


Promoted to 10th grade on time 


61.0 


NA 


Promoted to 12th grade on time 8 


40.7 


NA 






(continued) 



SOURCE: MDRC calculations from individual students' school records from the School District of Philadelphia. 



NOTES: Sample includes students from 22 nonselective, comprehensive high schools. Students in the sample 
were included on the district's transcript and attendance records. The sample excludes students who did not 
attempt at least one credit during a given school year. 

’’Typically, students who are overage for grade were retained in the current grade or a prior one. “Overage for 
grade” means a student turned 15 before the start of the 9th grade, 16 before the start of the 10th grade, 17 before 
the start of the 1 1th grade, or 18 before the start of the 12th grade. 
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Table 1 (continued) 



b Students were defined as repeating a grade if the district's administrative records indicated that they were 
enrolled in the same grade for both the current and the previous year. 
c State standards assessment test scores were available only for the 1 1th grade. 

d Attendance rates were calculated for each student by dividing the number of days the student was present by 
the total number of days the student was enrolled in a given year. 

e Until the 1998-1999 school year, students in the district were required to earn four course credits in order to 
be promoted. 

'For the purposes of this analysis. 9th-grade students were considered to have been promoted to the 10th grade 
on time if they were listed as lOth-graders in the next year's administrative data file. Discrepancies between the 
percentage of students meeting various promotion requirements and the promotion rate may be caused by 
students earning some credits in previous years, incomplete course-detail records, or inconsistent application of 
the promotion requirements. 

g For the purposes of this analysis, 9th-grade students were considered to have been promoted on time to the 
12th grade if they were listed as 12th-graders in the administrative data file three years after having finished the 
9th grade. 



percent of ninth-graders could be classified as chronic absentees (missing a total of more than 
seven weeks of school or an average of one day each week), while only 23 percent had atten- 
dance rates of 90 percent or higher. 

The typical ninth-grade student entered the district’s nonselective high schools with 
relatively low levels of reading and math skills. Based on SAT-9 achievement test scores from 
the eighth grade, the average ninth-grade student entered high school scoring at the 37th Normal 
Curve Equivalent (NCE) in reading comprehension and at the 35th NCE in math computation 
and problem-solving (the national average is the 50th NCE for both tests). 9 Among ninth-grade 
students, 23 percent entered high school with test scores at or above grade level in reading and 
1 7 percent entered with scores at or above grade level in math. 

Finally, Table 1 shows the rate at which ninth-grade students from the district’s schools 
were actually promoted to the tenth grade and were eventually promoted to the twelfth grade on 
time for their scheduled graduations. In all, fewer than two-thirds (61 percent) of ninth-graders 
were promoted to the tenth grade for the following school year. This percentage masks a more 
troubling pattern among students who had already repeated the ninth grade. Though not shown 
in the table, further analysis indicates that students repeating the ninth grade were much less 
likely to be promoted than students who were in the ninth grade for the first time. In all, 49 per- 



9 The Normal Curve Equivalent (NCE) is a way of measuring where a student falls along the normal curve. 
The normalized test score, which ranges from 1 to 99 with a mean of 50, allows for comparison across tests 
and subjects. Unlike percentile rank scores, the NCE measurement has an equal interval between scores, which 
means that NCE scores can be averaged to allow for comparison of groups of students or schools. 
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cent of the repeating ninth-graders were promoted to the tenth grade for the following year, 
compared with 66 percent of the first-time ninth-graders. 

Only 41 percent of the ninth-graders from the 1996-1997 through 1998-1999 school 
years were enrolled in a district public high school as twelfth-graders three years later, when 
they were scheduled to graduate. Again, the rates for those repeating the ninth grade (not shown 
in the table) were dramatically lower (16 percent) than those for first-time ninth-graders (51 
percent). 10 

Continued Problems in Upper Grades 

The second column of Table 1 presents information about the attendance, test score av- 
erages, and promotion status of tenth- through twelfth-grade students. In short, the data in Table 
1 suggest that even though these students were able to progress beyond the ninth grade, their 
success in high school was by no means guaranteed. Over 10 percent were repeating their cur- 
rent grade, and attendance rates averaged 77 percent (indicating that a typical student was ab- 
sent for an average of 45 days during the year — nearly the equivalent of a full marking period). 
Among students who reached the eleventh grade and took the state’s standardized assessment 
tests, only 10 percent performed at or above grade level for both reading comprehension and 
math. 



In short, the information in Table 1 indicates that ninth-grade students in the nonselec- 
tive, comprehensive high schools in the district are at high risk of leaving school with very low 
levels of reading and math skills. 11 The vast majority of entering students appear to be unlikely 
to earn a high school diploma. Talent Development is designed specifically to intervene in such 
situations and to focus as directly and intensively as possible on many of the root causes of 
these problems, which begin in the ninth grade and continue throughout the upper grades. 



The Talent Development Model 

Talent Development was conceived as a comprehensive paradigm for school reform, 
which asserts that all children can leam and will do so in an academic setting that is demanding 
and that expresses high expectations. From the beginning of Talent Development, its central 



10 The analysis is not able to track students who leave the district’s public schools. Students who are no 
longer enrolled in a district public high school may have dropped out of school or may have enrolled in a pub- 
lic high school in another district or in a private school. 

"The averages presented in Table 1 mask the variation among high schools in the district, some of which 
serve somewhat more affluent communities and enable somewhat higher percentages of their sUidents to make 
adequate progress through school. Talent Development aims specifically to serve students in the lowest- 
performing schools, many of which fall well below the averages presented in Table 1. 
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goals have been to help transform urban high schools into solid learning institutions that estab- 
lish a strong, positive school climate for learning; promote high academic standards for all stu- 
dents, especially in core subjects such as English and mathematics; and provide professional 
development to support implementation of the recommended reforms. This section reviews the 
theoretical rationale for the model, and describes its key operational components. 

Conceptual Framework 

Figure 1 presents a summary version of the conceptual framework describing Talent 
Development’s theory of change. This research-based theory identifies the problems Talent De- 
velopment attempts to address, specifies the model’s core elements, and defines the key goals 
and outcomes it is intended to accomplish. This is an idealized version of Talent Development, 
displaying all its intended attributes — but not always a reflection of how the model was actu- 
ally implemented. Its key purpose in spelling out the theory of change is to make explicit the 
pathways through which the core components of the model are intended to improve school 
functioning and, ultimately, student outcomes. 

The framework focuses on four successive and causally related phases in the Talent 
Development model; the hypothesis is that these phases, outlined below, will lead to improved 
student performance outcomes. 

• Structural elements: The concrete changes that Talent Development seeks 
to implement include changes in schools’ organization, policies, curriculum 
content, resource allocations, and relationships with external entities. These 
include five broad and mutually reinforcing elements: (1) reorganizing 
schools into small learning communities; (2) research-based curricula, de- 
signed to move all students toward advanced coursework in English and 
mathematics; (3) recovery opportunities and extra help for students who need 
it; (4) staff professional development systems designed to support implemen- 
tation; and (5) parent and community involvement activities that aim to en- 
courage students’ career and college development. 

• Supports and learning opportunities: Implementation of structural ele- 
ments in turn is expected to lead to improvements in school climate and func- 
tioning; positive changes in teacher and student behaviors, experiences, and 
expectations; and more productive use of internal and external resources. 

• Mediating outcomes: Enhancement of supports and learning opportunities 
are hypothesized to produce mediating outcomes, including improvements in 
students’ attitudes, levels of engagement, and sense of efficacy and 
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Simplified Conceptual Framework for the Talent Development High School Model 










competence that will enhance their willingness and capability to perform 
more effectively as students. 

• Performance outcomes: The mediating outcomes are then expected to lead 
to changes in performance, including positive changes in student achieve- 
ment, progress toward graduation, and preparation for successful transitions 
to postsecondary education and employment. 

These successive phases, again, represent an ideal trajectory, operating across the full 
range of possible interactions — one that is seldom completely experienced in practice. A more 
elementary and succinct version of Talent Development’s theory of change was presented by 
one of the model’s original designers: students find school more attractive and attend more fre- 
quently; they go to class, take and pass more courses, and tend to remain in school; as a result 
they are more likely to be promoted and eventually to graduate. 

How Talent Development is Designed to Address Key Educational 
Problems 

The structural elements listed above — essential building blocks of the Talent Devel- 
opment model on the ground — correspond to five key educational problems (shown in the top 
row of Figure 1) that Talent Development was designed to address. 

Key Problem 1 : Student Anonymity 

A reason often given by students for dropping out of high school is that they feel dis- 
tance and estrangement from teachers and administrators. 12 The large size of comprehensive 
high schools often depersonalizes the school environment, preventing teachers from working in 
teams or developing an atmosphere conducive to learning. 13 Continuous changes in classroom 
composition and student peer groups also increase anonymity and diminish students’ sense of 
community. 

To address this, Talent Development uses small learning communities as a means of 
building personalized relationships among and between students and teachers. The small learn- 
ing communities take the following forms: 

• Ninth Grade Success Academy. At the core of Talent Development restruc- 
turing is the Ninth Grade Success Academy, a self-contained school-within- 
a-school organized around interdisciplinary teacher teams that share the same 



12 Altenbaugh, 1998. 

13 Sizer, 1984; Hill, Foster, and Gendler, 1990; Powell, Cohen, and Fanar, 1985. 
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students and have common daily planning time. Practices and offerings are 
designed to help ease students’ transition into high school, encourage good 
attendance, and promote positive learning behaviors. 

• Career Academies. For students in grades 10 through 12, Talent Develop- 
ment high schools are organized into Career Academies, which are self- 
contained groups with their own management and instructional staffs located 
in a separate part of school buildings. They each are designed to enroll 250 to 
350 students and are organized around career themes. Career academies pro- 
vide all students with a core college-preparatory curriculum and work-based 
learning experiences supported by industry partners. 

Key Problem 2: Low Student Expectations 

Adolescent students become bored, and their attendance and interests suffer when they 
are not drawn to their class work by the prospect of challenging and fulfilling content, or when 
they see no connections between learning tasks and their own interests and future goals. Many 
students also may develop lowered expectations if they find themselves tracked into lower-level 
courses aimed at students not expected to attend college. For all students in low-performing 
schools, there are usually few opportunities to explore how classroom skills are actually applied 
outside the classroom. 14 

The Talent Development model uses both organizational and instructional reforms that 
aim to fight apathy by connecting schoolwork to students’ backgrounds, interests, and goals, 
and by enlivening lessons and learning activities with interesting and challenging applications. 
Relevant reforms include: 

• Refocusing the curriculum. As part of the Talent Development model, both 
the Success Academy and the Career Academies are designed to provide a 
curriculum that combines academic coursework necessary for graduation and 
for college admission (discussed further below) with a sequence of career- 
oriented courses and work-related awareness and development activities. 

• Providing a college-preparatory sequence for all students. Central to the 
idea of high expectations for all students is a shift away from tracking and a 
shift toward college preparation for all students. The Talent Development 
small learning communities achieve this in the following ways: 



14 See, for example. Resnick, 1987a; Raizen, 1989; Stasz et al., 1993; and Grubb, 1995. 
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• The Ninth Grade Success Academy, by doubling the course offerings in 
English and math, aims to prepare students to succeed in their core 
courses; in addition, the ninth-grade roster includes the required science 
and social science courses; 

• In the upper grades, Career Academies support a full four-year sequence 
in English and mathematics and offer science and social science courses 
at each grade level, along with career-pathway and other elective 
courses. 15 

• Creating extended class periods. Four 90-minute periods per day enable 
schools to use a variety of learning activities that call for students to work in- 
dividually and in cooperative teams on challenging and interesting topics. 

Key Problem 3: Poor Prior Student Preparation 

One of the greatest challenges secondary schools face is the wide diversity in the level 
and quality of preparation students receive prior to high school. The Talent Development model 
requires a common core curriculum for all students. It attempts to universalize standards-based 
education by providing increased academic learning time and significant recovery opportunities 
for struggling students. Elements of the high school model that attempt to address poor prior 
preparation and performance include: 

• Freshman Seminar. Offered during the first semester of the ninth grade, the 
Freshman Seminar provides a variety of techniques to help students develop 
studying, note-taking, and time-management skills, as well as the social rela- 
tions skills required in their academic setting and their lives outside of 
school. 

• Extended block schedule. Talent Development works with schools to create 
a “4x4” extended-period block schedule (that is, four periods per day for four 
courses each semester) which makes possible “double doses” of English and 
math in both the ninth and tenth grades, intensifying the effects of classroom 
instruction through extended, well-structured classes. 

• Catch-up courses. A key feature of the doubling-up of English and math 
courses are the ninth-grade catch-up courses that are offered during the first 
semester of both the ninth and tenth grades to prepare students to succeed in 



l5 Legters and Morrison, 1999, pp. 2.5 and 2.51; Center for Social Organization of Schools, 2002b. 



12 




district-mandated courses, required for graduation, which they take during 
the second semester. 

• Twilight Academy. An after-hours program is offered as an alternative to 
the regular school day for students who have serious attendance or discipline 
problems or who are returning to school from incarceration or suspension 
from another school. 

Key Problem 4: Limited School Capacity to Implement Comprehensive Reform 

Currently, most schools in highly stressed environments have little or no capacity to ad- 
dress the problems discussed so far: anonymity, low student expectations, and poor prior prepa- 
ration. Even with specific strategies, such as the examples given, implementing a comprehen- 
sive set of organizational reforms that respond to these challenges requires that teachers and 
administrators change their practices in fundamental ways. In light of this, two critical compo- 
nents of the Talent Development approach are sustained, multilayered, multiyear implementa- 
tion support and continuous professional development. These consist of: 

• A multiyear implementation plan. Talent Development staff work with 
schools to create a multiyear implementation plan for phasing in the key 
components of the Talent Development model. 

• An implementation support team. Each school is assigned a support team 
of Talent Development curriculum coaches, led by a school-based facilitator, 
who works with school leadership and other staff to implement the model, and 
curriculum coaches, who work with the teachers in the Success Academy 
teams and with other faculty in the schools. Ongoing technical assistance 
available to the schools could include teacher coaching, curriculum materials 
and other resources, workshops, and support for student-teacher meetings. 

• Professional development. Since 1998, annual national conferences for Tal- 
ent Development schools, and training institutes for their staff, have allowed 
the wider network of Talent Development schools and professionals to re- 
ceive training, to share their experiences, and to learn from one another. 

Key Problem 5: Schools’ Isolation from Families, Communities, and Local 
Institutions 

Many high schools in the district are isolated from other institutions in their communi- 
ties and have very limited contact with students’ families (often restricted to notification of se- 
vere disciplinary and academic problems). Little effort is made to use the community as a re- 
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source for providing students with meaningful learning opportunities and a context for high- 
lighting the potential relevance of what they are studying. 

To address this problem, Talent Development seeks to develop school-family- 
community partnerships by working with the National Network of Partnership Schools. 16 The 
goal is to enable families and communities to become informed about and involved in chil- 
dren's education and schools. Working together, Talent Development schools and facilitators 
from the Network employ a variety of strategies for increasing that involvement. 17 



Implementation of the Talent Development Model in Philadelphia 

This section discusses how implementation of the Talent Development High School 
model actually unfolded in high schools within the Philadelphia School District (hereafter, the 
district). Its major focus is to characterize the progress and effectiveness of Talent Development 
implementation and particularly its status during the 2003-2004 school year, when the field re- 
search took place. 18 

The implementation findings are presented in the sections that follow. However, two 
general points should be stressed. First, the Philadelphia expansion of Talent Development 
marked the first time that CRESPAR had taken the model to a new urban school district. While 
CRESPAR staff (and those of the local intermediary with whom they partnered) were experi- 
enced, knowledgeable, and skilled in working with schools, and did succeed in getting the 
model into seven high schools, the Philadelphia expansion, as a new enterprise, naturally pre- 
sented a series of challenges, stresses, and problems that were not all recognized by the imple- 
mented before they started. Understandably, the implementation process felt the effects of this. 

Second, Talent Development’s expansion in Philadelphia lacked the formal participa- 
tion of the school district, a point whose positive effects are discussed immediately below. 
However, it seems fair to speculate that direct district support might have led to greater stability 
at the school leadership level (as it has for reform efforts in other districts); a more authoritative 



16 Center for Social Organization of Schools, 2002a. 

17 It should be noted that efforts to launch community partnerships were not undertaken in Talent Devel- 
opment schools in Philadelphia until the 2003-2004 school year, and had little visible effect on the quality of 
implementation. 

18 The field research consisted of interviews with principals in five of the seven Talent Development 
schools, which were implementing the model in the 2003-2004 school year; interviews with most Talent De- 
velopment staff, including the director, the staff teams in the schools, other staff at the Center for Research on 
Students Placed At Risk (CRESPAR), the Philadelphia Education Fund (the local intermediary that shared 
implementation responsibilities with CRESPAR), and the School District of Philadelphia; review of existing 
reports, records, school schedules and other material; visits and observations at all of the schools, as well as 
attendance at professional development seminars in both Philadelphia and Baltimore. 
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(and less negotiated) role for the implementers; more consistent and predictable funding; greater 
technical support; and more timely assignment of teaching and other staff to the Talent Devel- 
opment schools — in short, to a better-supported implementation experience. 

What instead happened was that the implementation process in Philadelphia was ex- 
posed to (and not shielded from) the effects of contextual factors normally found in large urban 
school districts: policy vagaries, inconsistencies, bureaucratic inflexibility, staff turnover, and 
organizational turbulence. Scheduling, resources, personnel, and degrees of participation by 
schools all were to some degree affected. It is, of course, not possible to gauge with precision 
how much of an effect this had. Yet any limitations in the success or variation in the quality of 
implementation need to be interpreted in light of the probable effects of the hands-off policy 
regarding Talent Development that the district chose to adopt. 

Implementation Findings 

• Implementation of Talent Development was sanctioned by the district but 
received neither formal endorsement nor direct institutional support. 

By 1998, CRESPAR had begun looking for venues where it might expand the Talent 
Development model that had been created in Baltimore. Nearby Philadelphia represented one 
intriguing option. The superintendent of the School District of Philadelphia at that time had pre- 
viously been an educational leader in Maryland and had personal relationships with CRESPAR 
staff. His then-primary emphasis in the district was reform efforts directed at younger children 
and lower grades. There was far more limited interest on the part of the district in the high 
schools; indeed, there was at that time no distinct office of secondary education in the district. 

The superintendent thus positively greeted CRESPAR' s proposal to introduce Talent 
Development into high schools in the city. However, the agreements that resulted were in many 
respects quite informal. The district would agree to sanction Talent Development's presence in 
the schools. This in fact fit in well with a district mandate (which was somewhat laxly enforced) 
that all failing high schools adopt some reform model. The district would also provide funding 
and support, on a school-by-school basis, for introduction of Talent Development (even as it 
would support other reform models). 

But these agreements were not expressed in official contracts or agreements. There was 
no recognition by the district of Talent Development as a “model of choice” for improving high 
schools in Philadelphia. Nor was formal or clear institutional support provided by the district for 
implementation efforts on behalf of Talent Development; no written agreement between the 
district and CRESPAR was ever signed. 
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In the end, though, Talent Development did come to be implemented in schools in the 
district. The lack of official support notwithstanding, informality proved beneficial in the short 
run. It afforded latitude to the implementers to identify and work with individual high schools 
without the need to develop formal protocols or get procedural approvals from the district for 
how those contacts would be made. Though there were naturally some limitations, the imple- 
mentation staff had something of a free hand in how it could operate, which schools it could 
choose to work with, and how it would carry out planning and implementation. 

• Talent Development implementation in Philadelphia was jointly carried 
out by CRESPAR and a local educational intermediary. 

To facilitate its work in Philadelphia, CRESPAR developed a collaborative relationship 
with the Philadelphia Education Fund (the local intermediary), a non-profit organization whose 
mission is improvement of education in the city, especially for low-income, underachieving 
young people. The local intermediary’s capacities included research, advocacy, program design, 
and direct involvement in schools. It had worked collaboratively with the district for many 
years, was conversant with local school issues and district leadership, and had credibility in the 
local school community. 

The partnership that emerged was informal. Staffing for the Talent Development effort 
was often shared between the two organizations, and the roles and interactions among 
CRESPAR, the Talent Development schools, and the local intermediary were extremely fluid, 
though again direct involvement with the district was limited. The funding agreements that un- 
derlay this setup were complex, and similarly informal. In all, the estimated additional cost of 
operating Talent Development is approximately $250 to $350 per student per year, which in- 
cludes materials, technical assistance, and salaries for curriculum coaches and a part-time pro- 
gram facilitator. Four major sources of funding were used to underwrite the costs of the Talent 
Development initiative: 19 

1 . Federal grant funds that supported Comprehensive School Reform Projects 
in the district; 

2. Federal grant funds that supported Small Feaming Communities in the district; 

3. Funds available to CRESPAR that supported staffing, training and profes- 
sional development, curricula, and other expenses related to the Philadelphia 
initiative; 



19 The costs discussed here are the incremental costs of implementing Talent Development, over and above 
the basic budgets that individual schools would receive based on their school enrollment and normal share of 
such special funding streams as Title I. 
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4. School district funds provided to the local intermediary for support of im- 
plementation activities beyond what the two federal sources would support. 

In addition, the district provided additional teachers to the participating Talent Development 
schools needed to implement block-scheduling and team teaching (the latter described below); 
though supplied as staff, this was in effect an additional financial contribution by the district to 
the Talent Development effort. 

The grant proposals for the federal funds were prepared on behalf of the participating 
high schools by CRESPAR and the local intermediary and formally submitted to the funding 
sources by the district. Through the 2003-2004 school year, the federal grant and district funds 
were also largely administered on behalf of the schools by the local intermediary, with limited 
input on the part either of the district or the schools. 20 

What this meant was that the implementers could exercise considerable flexibility and 
creativity in going about their work. Given the informality of support from the district, this also 
meant that adequate funding for Talent Development in Philadelphia was a somewhat uncertain, 
year-to-year issue. However, even though a predictable and reliable base of support for Talent 
Development’s continuation never was established, adequate funds to support the implementa- 
tion process were available through the 2003-2004 school year. 

• An evolutionary process that did not directly involve the school district, 
the recruitment of Talent Development schools was guided more by the 
interest of individual schools than by use of objective selection criteria. 

There was no formal procedure for identifying and recruiting what eventually became the 
Talent Development schools. The issue of which schools and how many would participate ap- 
pears not to have been addressed upfront in any systematic way. There was, of course, a guiding 
precept: Talent Development would seek out low-performing schools in the district since its intent 
and elements were directed toward changing outcomes in schools of that kind. But objective crite- 
ria for identifying and selecting the schools that eventually participated were never used. 

Failing high schools in Philadelphia had been encouraged to seek out and adopt a re- 
form model. They had latitude in which model they could adopt. Talent Development staff, 
having made some presentations and contacts in the city and at educational conferences, be- 
came known to several high school principals who had heard about the model and wanted to try 
it in their schools. Indeed, word-of-mouth was an important recruitment tool for the program. 



20 Beginning in the 2004-2005 school year, participating schools took a much stronger role in determining 
how the funds would be programmed and used. 
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School principals usually reached out to Talent Development, rather than the other way around. 
There was no direct involvement by the school district in this process. 

Recruitment of additional of Talent Development high schools thus took place on an ad 
hoc basis, without an overall plan for ramp-up or incorporation of new schools. By the start of 
the 2003-2004 school year, a total of seven schools were actively implementing the Talent De- 
velopment model. (An eighth school, School F, began to close in the 2003-2004 school year 
and, thus, is not included in the analysis.) These schools had been brought into the process over 
a total of four years. Table 2 displays the sequence of implementation. 

Adoption of Talent Development was designed to be a two-stage process. After initial 
interest was expressed, a high school would engage in a “planning year.” During this period, 
with assistance from CRESPAR and the local intermediary, school staff would familiarize 
themselves with the Talent Development model, in some cases visiting established Talent De- 
velopment schools in Baltimore; develop a plan for implementing the model successfully in 
their school; and, after a formal vote by faculty, adopt the model. 

It should be noted that the last two schools to join the Talent Development complement 
were only in their first year of implementation when the field research was undertaken, and 
when the final impact data were obtained. Therefore, while they figure in the general discussion 
that follows, their experience is not heavily weighted, nor are they included in the analysis of 
impacts later in this report. 

• There were variations in the intensity and quality of implementation 
across the Talent Development schools. 

Though Table 2 might suggest an adoption process that was repeated with consistency 
and uniformity in each school, this was not the case. Several factors combined to produce dif- 
ferences in the smoo t hness and intensity of the process in the seven high schools that came to 
adopt Talent Development. 

First, the expansion process, especially in the early going, was evolutionary in nature. 
Its major steps were adapted from the experience in Baltimore, but the adaptation happened in 
real time — chiefly in the first two Philadelphia schools that initially chose to adopt. These 
schools benefited from the nearly full-time presence of an implementation manager who had 
had direct experience in the Baltimore pilot schools. But the rollout in these two schools was a 
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The Talent Development Evaluation 
Table 2 

Implementation Time Line in Eight High Schools 



School A 
School B 
School C 
School D 
School E 



1998-1999 



Planning 



Planning 



1999-2000 



Implementation 
Year 1 



Implementation 
Year 1 




School F 
School G 
School H 



School Year 
2000-2001 

Implementation 
Year 2 



Implementation 
Year 2 



Implementation 
Year 1 




2001-2002 


2002-2003 


2003-2004 


Implementation 
Year 3 




Implementation 
Year 4 




Implementation 
Year 5 








Implementation 
Year 3 




Implementation 
Year 4 




Implementation 
Year 5 








Implementation 
Year 2 




Implementation 
Year 3 




Implementation 
Year 4 








Implementation 
Year 1 




Implementation 
Year 2 




Implementation 
Year 3 








Implementation 
Year 1 




Implementation 
Year 2 




Implementation 
Year 3 








(^^Planning^^) 


Implementation 
Year 1 


NA 



Planning 



Implementation 
Year 1 



Planning 



Implementation 
Year 1 












new experience, even though CRESPAR moved quickly to codify the planning process it had 
developed for use in subsequent district schools wishing to adopt the model. 

Second, adopting schools were attracted to Talent Development for different reasons. In- 
terviews suggest that principals in the earliest two schools to adopt the model clearly wanted to 
reform the overall high school. Wi t hin those schools there was a greater effort to make changes 
that would affect all grades. In others, the interests were somewhat less strategic, much more fo- 
cused on addressing the needs of ninth-graders through installation of the Success Academy, and 
less so with the broader conception of school reform espoused by Talent Development. 

Third, the complement and mix of Talent Development staff deployed by CRESPAR 
and the local intermediary to coordinate the implementation changed over time. Especially in 
the early years of implementation, when several schools were in the planning or early imple- 
mentation phase, the staff who were implementing the program were stretched thin, often hav- 
ing to respond to needs in two schools at once. After the first two schools had begun implemen- 
tation, CRESPAR and the local intermediary were able to deploy staff from those schools who 
had worked through the implementation process directly. Two such staff members, in fact, 
managed implementation at later-implementing schools and also worked as trainers for 
CRESPAR. 

Fourth, the availability and “take-up” of training and professional development varied 
among schools. Teachers needed to become conversant with the Talent Development approach, 
the Talent Development curriculum, and also the use of the extended block schedule — for 
many a considerable change from their past experience and practice. While both CRESPAR and 
the local intermediary offered numerous professional development opportunities (as described 
below), participation by teachers was uneven. 

School-specific factors sometimes played a part in implementation quality. At one 
school, Talent Development was adopted even though the faculty voted against the project; at 
another, Talent Development started up (that is, the Success Academy was installed) before the 
planning process was completed and faculty had voted (they did ultimately endorse implemen- 
tation of the model). The quality and coherence of existing Career Academies and small learn- 
ing communities also varied among (and within) the schools. 

Finally, it should be noted that principal turnover was evident during the implementa- 
tion process. Of the first five implementing schools, four experienced changes in leadership; in 
two cases, the change occurred within a year of the program having been adopted at the princi- 
pal’s behest . 21 These changes did not always affect Talent Development negatively — two of 



21 In only two of the seven schools that implemented Talent Development had the same principal been in 
place throughout the implementation process. 
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the successor principals were strong supporters of the program — but they did add a degree of 
uncertainty and turbulence to the adoption process. 

• The Success Academy was the most strongly and consistently imple- 
mented set of elements of the Talent Development model. 

The Talent Development strategies for addressing ninth-grade issues, embodied in the 
Success Academy, had five main features as implemented: 1) a restricted physical setting, in 
which the needs of the incoming freshman class could be met, and support provided, in a dis- 
traction-free, concentrated way; 2) a team-teaching structure designed to break the class into 
smaller, more intimate groups, identify specific youngsters needing assistance, and provide that 
assistance effectively; 3) a supportive environment that would encourage students to attend 
school regularly and achieve academically; 4) a curricular regimen built upon the extended 
block schedule, which was designed to help students overcome skill and knowledge deficien- 
cies; and 5) the Twilight Academy, a specialized program for ninth-graders who failed, or ex- 
perienced difficulty, in the normal school setting. The manner of their implementation is de- 
scribed in the following sections. 

Setting 

In all of the schools, a separate floor or wing of the school was designated as the Suc- 
cess Academy, with its own marked entrance wherever possible (not all schools had physical 
layouts that would allow this) and signs designating the area as the Success Academy. And 
while this physical rearrangement of space was an obvious logistical challenge, all of the Talent 
Development schools achieved it. This setup contrasted markedly with that of a traditional 
school in the district, where entering ninth-graders would join the entire student body and where 
their courses might be held anywhere in the building. 

The incoming ninth-grade class in Talent Development schools was divided into three 
or four separate groups (depending on the size of incoming enrollment), usually of about 90 
students each. An Academy Principal, with release time, directed the overall effort. The Talent 
Development implementation team (organizational facilitator and coaches, discussed further 
below) also was almost always located in the Success Academy floor or wing, helping to add to 
an environment that offered more personalized attention to the freshmen. This again contrasted 
with a traditional school setting, where ninth-graders were neither kept in small groups nor fo- 
cused on distinctively as a freshman class. 

Team-Teaching 

Teaching these small groups of Talent Development ninth-grade students was the re- 
sponsibility of teaching teams. Each team had a Team Leader who coordinated the team’s work, 
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handled discipline problems that spilled out of classrooms, and was to receive a reduced teach- 
ing load. The rest of the team was comprised of a group of teachers who were to stay with their 
student group throughout the academic year. In addition, class schedules were set to ensure that 
the teams would have common planning times in which they could meet to discuss issues 
among their students, resolve disciplinary problems (which might involve meeting with students 
and their parents), and address curricular or teaching issues. 

The structure generally worked, although the ideal team structure appears to have been 
achieved and continuously maintained in only one of the Talent Development schools. In oth- 
ers, the setup was achieved but not sustained, for two reasons. One was the difficulty in coordi- 
nating the schedules of each team staff (so they could meet regularly as a group), as well as en- 
suring that each team consisted of the right teachers for the ninth-grade curriculum. The exper- 
tise to handle the scheduling and rostering of students and staff was not always available in the 
Talent Development schools. The second reason was that the resources (monetary or teaching) 
to support the required release time for team leaders were not always made available. In those 
cases, the team leader role was not truly established, and the leader’s normal responsibilities had 
to be shared among other teachers. 

A related issue affecting the success of teaching teams was teacher experience. Experi- 
enced teachers were often reluctant to teach in the Success Academy because they did not want 
to deal exclusively with the challenge of teaching ninth-graders. Individual Talent Development 
schools struggled with the tension between assigning more experienced teachers there — 
where, arguably, their skills were more needed — or instead staffing the Success Academy with 
less experienced, or novice, teachers. 22 

Supports for Students 

The Success Academy also made use of incentives and recognition programs to en- 
courage regular attendance at school. Prizes, pizza parties, and award ceremonies were directed 
at students who achieved perfect (and/or 90 percent) attendance during a given month, and also 
for students with high grades. Large attendance charts were created and posted throughout 
school hallways to reinforce the message that attending school was important; names of stu- 
dents with perfect attendance and outstanding grade performance were also displayed through- 
out the Academy space. 



An one school, the principal made the plausible decision to reassign some of the more experienced staff, 
who had been in the Success Academy, to teach more eleventh-grade classes, reasoning that that might help 
boost scores on the Pennsylvania System of School Assessment (PSSA) test — the high-stakes test used to 
determine Adequate Yearly Progress under the federal No Child Left Behind Act. 
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Another key component of the Success Academy was the regular use of “report card 
conferences.” Small teams (sometimes supplemented by staff from CRESPAR or the local in- 
termediary) would meet with each student each time report cards were issued, review the stu- 
dent's grades, help the student assess progress toward promotion, and provide encouragement 
and support. This also became an opportunity for students to meet with individual teachers 
when they were experiencing problems with one of their courses. 

Specialized Curriculum 

The academic centerpiece of the Success Academy was the combination of extended 
block scheduling, “double-dosing” of key subjects and the CRESPAR Talent Development cur- 
riculum. CRESPAR’ s curriculum was designed to let students catch up from low performance 
levels commonly found when they entered high school in the ninth grade and meet the academic 
requirements of subjects — math and English in particular — that they would be taking there. 

The typical ninth-grade curriculum in a Talent Development school is summarized in 
Table 3. As the top half of this table shows, the first semester of ninth grade was pivotal in 
achieving the supportive academic “bridging” in the Success Academy. Both Transition to Ad- 
vanced Mathematics and Strategic Reading were designed as preparatory “catch-up” courses 
that would enhance the skills of incoming freshmen and enable them to succeed in traditional 
ninth-grade algebra and English. Freshman Seminar, combining study skills, personal goal- 
setting, and social and group skills, was designed to prepare students more broadly for the de- 
mands of high school. In practical terms, this meant that ninth-graders were routinely scheduled 
to take algebra in Talent Development schools, whereas in traditional schools, students with 
inadequate background might take a year of a lower-level course during ninth grade, and then 
take algebra later. 

The success of this course arrangement in the Success Academy rested on extended 
block scheduling. 23 In Talent Development schools, extended block scheduling worked as fol- 
lows: Students were scheduled to take four courses per semester, each meeting for 90 minutes 
per day. Each one-semester course was worth a full credit toward graduation. By contrast, stu- 
dents in traditionally rostered schools would take six courses, each lasting two semesters and 
each worth one credit. The contrast between the two rostering arrangements is reflected in a 
comparison of the top and bottom sections of Table 3. The extended block schedule permitted 



23 Though extended block scheduling was a critical feature of the Ninth Grade Success Academy, Talent 
Development schools in Philadelphia followed this schedule in all grades. 
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The Talent Development Evaluation 
Table 3 



Typical Ninth-Grade Curriculum in Talent Development and 
Non-Talent Development High Schools 



Talent Development High Schools 


First Semester 3 


Second Semester 


Transition to Advanced Mathematics 


Algebra 


Strategic Reading 


English 1 


Freshman Seminar 


Social studies/history 


Science 


Elective 


Non-Talent Development High Schools 


First Semester 


Second Semester 


English 1 


English 1 


Mathematics (algebra) b 


Mathematics (algebra) 


Social studies/history 


Social studies/history 


Science 


Science 


Physical education/health 


Physical education/health 


Elective 


Elective 



NOTES: 

“The first three courses in this box are CRESPAR curricula, regarded as electives by the district. All other 
courses for Talent Development and non-Talent Development schools are district courses. 

'’Entering 9th-graders with adequate math background took algebra; others were assigned to general 
mathematics courses. 



Talent Development to offer more concentrated instruction and more opportunities to earn 
course credits. 

Cumulatively, this arrangement offered potential advantages to students in Talent Devel- 
opment schools: over a four-year period, they could potentially complete 32 credits, compared to 
24 for students who attended schools with traditional rosters. It also permitted some flexibility in 
rostering students who failed courses and needed to repeat them, especially if the roster chair — 
the school staff person responsible for setting up the overall roster and schedule the school fol- 
lowed — was knowledgeable about how to use block-scheduling effectively. For ninth-grade stu- 
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dents in Talent Development schools, though, there was the opposite of flexibility; double-dosing 
and the CRESPAR curricula effectively locked their course choices for the first year. 

By and large, extended block scheduling worked smoothly and was well-regarded by 
teachers. All of the schools were successful in putting it into place; in fact, three of the seven 
Talent Development schools either had implemented it before Talent Development or had pre- 
viously experimented with it. 24 Since the CRESPAR courses were developed explicitly for use 
in that arrangement, and were also well-supported with printed materials and handouts, teachers 
found them easy to deliver. 

Teachers generally judged Strategic Reading and Transition to Advanced Math to be of 
adequate quality and rigor. There was somewhat less agreement regarding Freshman Seminar. 
Taught mainly by social- science teachers (who in many cases would follow it in a second se- 
mester with their own history course), the Seminar was sometimes characterized as too “soft” 
and lacking in content. Some teachers taught only parts of it or began, late in the first semester, 
to work social science and history into the content. Not surprisingly, as implementation pro- 
ceeded, other adaptations to the CRESPAR curriculum crept in. In one school — whose princi- 
pal was a former English teacher — Strategic Reading in the ninth grade was dropped. In an- 
other, the course was adapted to include elements of the second-semester English I course. 

One additional point should be noted: extended block scheduling, to work most effec- 
tively, requires teachers who have had training in how to use it well, dividing 90-minute seg- 
ments into engaging and well- structured sub-units and activities. Otherwise — as teachers and 
administrators alike noted — blocked classes could become monotonous time-wasters for stu- 
dents. While Talent Development coaches were available to assist teachers, not all of them took 
advantage of that help; nor did teachers new to Talent Development schools routinely receive 
training in use of the block. Thus teacher turnover, which occurred in each of the Talent Devel- 
opment schools, added to the challenge of maintaining instructional quality in the extended 
block-roster setting. 25 

The Twilight Academy 

The Twilight Academy was designed as a special program for current and “repeater” 
ninth-graders who either needed special academic support or needed (for disciplinary or other 
reasons) to be placed outside the normal school environment. This program usually operated in 



Indeed, the block schedule, though not widespread, was already in use in a number of high schools in 
Philadelphia. 

25 In addition, during the Talent Development implementation period, the school district began to develop 
its own plans for high schools, including a standardized curriculum (discussed below), which led to further, ad 
hoc, adaptations of the idealized Talent Development course arrangement. 
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a separate section of the school and outside normal school hours — often later in the day so that 
some students could work or attend to family matters. 

All but one of the Talent Development schools succeeded in putting this element of the 
model in place. The Twilight Academy was an important complement to the Success Academy. 
It provided flexible and tailored help to students who were struggling academically. They could, 
after they had completed missed work or failed courses, rejoin the main school. Just as impor- 
tant, though, was the Twilight Academy’s value in providing a setting where potentially disrup- 
tive students could be placed, in lieu of suspending/expelling them or transferring them to one 
of the district’s disciplinary schools. 

Overall, it seems clear that the Success Academy, embodying Talent Development’s 
strategies for ninth-graders, was the most strongly implemented element of the model in the 
Philadelphia schools. It achieved its major purpose: creation of a separate and supportive envi- 
ronment for ninth-graders that intensified the amount of attention they received, both academi- 
cally and personally. Not all the pieces were equally well-established or maintained, but the ba- 
sic structure was implemented and positively viewed by principals and teachers. 

• Talent Development had limited success in transforming the upper 
grades into Career Academy programs fully supportive of the model. 

The Career Academies were perhaps the most problematic element of the Talent Devel- 
opment model in Philadelphia. In the Talent Development context, these were intended to extend 
concepts of the Success Academy into the upper grades: small learning communities, each aca- 
demically challenging, each with one or more career themes that would permit continued personal 
support to students, even in large high school settings, and prepare them for college. 

In practice, they encountered a prior history of similar structures in Philadelphia, all of 
which were responding to changes in thinking on the part of the district. Before Talent Devel- 
opment, Philadelphia had a long history both of small learning communities and of Career 
Academies throughout its comprehensive high schools; indeed, several of Philadelphia’s acad- 
emies became models for national adoption. 

Thus, in all the Talent Development schools, there already existed academy-like entities 
of varying quality and emphasis. Some schools had coherent and effective academies; in others 
they were weaker, and the quality of academies likewise varied within individual schools. Some 
of the small learning communities had consciously adopted career themes; others had not. Some 
were designated for higher-achieving, college-bound students. And some of what were desig- 
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nated as Career Academies were in fact not very different from traditional vocational education 
programs, and they often lacked the academic rigor of the ideal academy program. 26 

Changing the existing patterns proved a difficult challenge for Talent Development’s 
implementers. Their task in Philadelphia was not to create new structures, de novo, as was the 
case with the Success Academy. Instead, their challenge was to recast a collection of academy- 
like entities that had their own histories and in some cases their own rigidities as well. The 
changes desired by Talent Development staff in the existing academies were often subtle ones 
of philosophy and quality, which can be more difficult (and sensitive) to articulate and take 
longer to achieve. Schools who adopted the Talent Development model could, not without some 
justification, view the Career Academy component of the model as something they already had 
in place. It is likely, therefore, that the upper-grade experience of students in Talent Develop- 
ment schools did not greatly differ from that of students in non-Talent Development schools. 

Complicating all this was an initiative by the school district to adopt formal criteria for 
designation of Career Academies. During the 2003-2004 school year, the district began, to re- 
quire that, in each Career Academy, at least one faculty member have private-sector experience 
relevant to its theme in order for it to be officially recognized as an Academy. Talent Develop- 
ment schools (as well as other high schools in the district) thus were somewhat more concerned 
with working out which of their existing academies could be so designated than they were with 
aligning them with Talent Development standards. 

By and large, therefore, the upper-level academies were only weakly associated with 
the Talent Development model. The existing academies (staff often referred to them inter- 
changeably as academies and small learning communities) did not as a rule regard themselves 
as an element of Talent Development, which they often equated narrowly with the Success 
Academy. This was especially true in the eleventh and twelfth grades, which lacked both dou- 
ble-dosing of key subjects and any CRESPAR curriculum. 27 These upper grades had, in addi- 
tion, markedly less contact with Talent Development staff and coaches. 

Ninth-graders in the Success Academy were offered the opportunity to choose which 
upper-grade Academy they preferred to enter; first choices were honored as much as possible. 
Individual Academies offered orientation sessions and assemblies in which the career emphases 
and other features of the Academy were described. Success Academy faculty and the Talent 
Development team in each school helped plan and coordinate that effort. 



26 In one school the academies were reorganized in an effoit to eliminate implicit tracking, and to equalize 
their academic quality. The end result, whatever its effects on academic quality, produced a group of academies 
that had only limited coherence in terms of career themes. 

Fkms for the production of eleventh-grade “double-dosing” curriculum were not realized, though two 
schools in Philadelphia piloted a version of an eleventh-grade math curriculum produced by CRESPAR. 
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One important variation to the Talent Development model in the upper grades should 
be noted: the creation (in four of the seven Talent Development schools) of a “Tenth Grade 
Academy.” Two main reasons were cited for this variation. First, the evident effectiveness of 
the Ninth Grade Success Academy encouraged schools to extend the structure for another year. 
Part of this thinking appears to have been prompted by some increase in disciplinary problems 
observed among sophomores. 28 The second consideration, voiced by one principal, was that a 
more structured tenth grade might help boost students’ performance on the standardized test 
they would take the following year. 

The Tenth Grade Academy was far less structured than its ninth-grade counterpart and 
still in a fairly evolutionary state. Extended double -dosing and the CRESPAR tenth-grade cur- 
ricula were used in all the Talent Development schools. In only one school were sophomores 
located apart from other students. 

The potential difference in schools using a Tenth Grade Academy was that Career 
Academies had to be set up somewhat differently. The arrangement adopted by one school was 
that students made a choice of upper-level Academy in ninth grade but did not actually begin a 
structured sequence of Academy courses until eleventh grade. 

• The presence of a Talent Development implementation team in each high 
school was instrumental in getting and keeping the model established. 

To make Talent Development work, a small implementation team (paid for centrally 
through the local intermediary and CRESPAR, rather than directly out of individual school 
budgets) was deployed in each of the schools. Its head was the Talent Development organiza- 
tional facilitator, whose broad mission was to ensure that Talent Development implementation 
progressed smoothly and steadily within the school. In addition, each team was to have curricu- 
lar coaches (for literacy, math, and freshman seminar), who would assist teachers in making 
effective use of the block schedule and in using the CRESPAR curricula successfully. 

Putting these staff complements into place was the responsibility of CRESPAR and the 
local intermediary. Especially in the early implementation years, the staffing patterns were evo- 
lutionary, as would be expected with a new and expanding program pressed to find and deploy 
qualified staff. It was not until the 2003-2004 school year that all Talent Development schools 
came close to having the staff complement that had been anticipated. 29 



28 This may have been due, in part, to the increased numbers of ninth-graders who reached the tenth grade. 
29 Even in that year, organizational facilitators did not work full-time in some cases; some coaches split 
time between schools. 



28 




The organizational facilitator coordinated the Talent Development-related activities in 
the school: providing overall support for the Success Academy; ensuring that curricula and 
other materials were in place; overseeing the time and efforts of coaches; arranging report card 
conferences and incentive and award programs; and liaising with the principal and other senior 
staff in the schools. 

The coaches worked one-on-one with teachers, helping them to use the Talent Devel- 
opment curriculum, modeling teaching approaches, sometimes team-teaching with teachers, and 
helping to support the classroom activities of teachers generally, especially in their use of the 
block schedule. In some cases, they also worked with small groups of teachers and provided 
help to individual students. 

The use of coaching resources by faculty was entirely voluntary. Individual teachers 
were free to refuse offers of assistance and to refuse coaches entry into their classrooms; and 
some did. In addition, because CRESPAR curricula were limited to the lower (ninth and tenth) 
grades, so too by and large were the efforts and interactions of the Talent Development coaches. 
Uniformly, they reported only limited contact with upper-grade teachers. However, teachers at 
all grade levels who did make use of the coaches’ advice, knowledge, and training generally had 
high praise for their contributions. 

From one perspective, then, the roles of this core Talent Development team were clear: 
to solidify implementation of the model in the school, to engage school leadership in making 
needed changes, and to provide ongoing technical support, enrichment, and new pedagogical 
strategies to teachers in the school — particularly to teachers using the CRESPAR curricula. 
The team was there, in that sense, to bring about change within the school. 

In practice, the roles were more diffuse. The Talent Development implementation team 
members had no mandate “from above” to aid them within their school. The purview and au- 
thority of team staff derived not from the school district (whose support for Talent Development 
was distant and unofficial) but rather from the principal. Principals, though they approved gen- 
erally of Talent Development, quite understandably also viewed its implementation team as an 
open-ended resource to the school. The team members, in turn, recognized that they could not 
unilaterally set their own agendas but needed to support the principal’s priorities. 

Thus Talent Development’s in-school team was heavily involved in standardized test- 
ing — a substantial and time-consuming priority in all the district high schools. Both organiza- 
tional facilitators and coaches assisted with encouraging students to attend and take the test, 
with test prep, with follow-up testing, and with other logistical support. They were involved in 
identifying and setting up extra tutoring for eleventh graders who were expected to perform well 
on the state test. They assisted with efforts to identify and assist seniors who were “on track” to 
graduate on time. 
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Coaches sometimes represented Talent Development schools at training sessions man- 
dated by the district, in support of its new curriculum. They then returned to train teachers 
within the schools. Organizational facilitators and coaches sometimes prepared and carried out 
training sessions during district-mandated professional development periods. They also some- 
times dealt with disciplinary “pullouts” — students whose presence in a class was disruptive. 

These activities did not conflict directly with Talent Development. But they did reflect 
the tension between the implementation team’s putative role as “change agent” and the natural 
tendency of the school to view its staff as building resources (a view the Talent Development 
staff themselves understood and responded to). They also diluted the energies that Talent De- 
velopment staff could put to enhancing the “whole-school” nature of the model. While the staff 
was known to the entire school, the bulk of its interactions (not counting relationships with the 
principal) was with the Success Academy staff; coaches, as noted, found their efforts concen- 
trated on ninth-grade teachers and those tenth-grade teachers using CRESPAR curricula. 

• Professional support for Talent Development schools and staff was ade- 
quate and supported the implementation process, but uneven utilization 
of training opportunities limited its overall effectiveness. 

Beyond the role of the coaches within the schools, both CRESPAR and the local interme- 
diary had, from the beginning of the implementation process, ensured that opportunities for pro- 
fessional development would be available to the staff in Talent Development schools. CRESPAR 
was instrumental in arranging visits for staff from potential Talent Development schools to sites in 
Baltimore already implementing the model. It also instituted a set of summer training institutes to 
prepare Talent Development staff, coaches, and faculty for the challenges ahead. 

Likewise, for staff and teachers at Talent Development schools, the local intermediary 
operated a series of training and orientation sessions held in Philadelphia in late summer, deal- 
ing with block scheduling, the use of CRESPAR curricula, and the roles of coaches and other 
Talent Development staff. The local intermediary and CRESPAR technical staff were also 
available by phone and e-mail to help with specific issues, and monthly technical assistance ses- 
sions were also held . 30 

Teachers and staff who participated in any of these sessions found them to be relevant, 
constructive, and effective in preparing them for their work in the context of the Talent Devel- 
opment model. What limited the impact of these professional development opportunities was 



30 There was some reduction in these latter activities during the 2003-2004 school year, due primarily to 
changes in staffing arrangements between PEF and CRESPAR. 
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that participation was inconsistent and limited, especially after the first two schools had imple- 
mented the model. There were several reasons for this. The most difficult-to-address issue con- 
cerned the district’ s routine for assigning new teachers to schools. As a rule, the staffing deci- 
sions were not made until summer (often late in the summer). As a result, teachers assigned to 
Talent Development schools (especially novice teachers) might not be informed of their as- 
signments until after Talent Development training took place. 

In addition, availability of funding to defray the cost of attendance at training sessions 
varied from year to year. It was sometimes provided through the local intermediary and 
CRESPAR and sometimes by the principals in their own budgets. Given teachers’ (and admin- 
istrators’) summer schedules, which often conflicted with the schedule of trainings, the level 
and mix of attendance would also vary. 

The result was that training and professional development were unevenly distributed 
among staff in the Talent Development schools. The schools in the early implementation wave 
appear to have had far higher participation in professional development activities. Among the 
later-implementing schools, there were more venues for training, but neither the impetus for 
participation nor the funding was as strong or consistent. And the structural problem — teachers 
assigned to Talent Development schools too late to be able to participate in professional devel- 
opment — was a consistent problem. 

• Though special settings and support for repeating ninth-graders were 
planned, these were never consistently implemented in Philadelphia. 

The mainstream Talent Development strategies for ninth-grade students were created 
largely to address the needs of first-time ninth-graders. Repeaters represented a special problem. 
They might have been in the school building before Talent Development’s onset; they might 
also have a range of educational and social needs that the mainstream Talent Development 
model would not address. 

The basic plan was for repeaters to be kept in distinct classes and for specialized assis- 
tance to be provided for them. In practice, none of the Talent Development schools achieved 
this. Schools instead looked at the overall course and promotion status of these students. Stu- 
dents who might be missing just one or two courses presented a different challenge than stu- 
dents who had failed three or four courses. As a consequence, repeating students found their 
way to three more or less distinct settings. They might be placed in Twilight Academy, the spe- 
cially designed setting for students experiencing academic or disciplinary problems. Alternately, 
they might be placed into the overall ninth grade, perhaps retaking courses they had failed and 
needed for advancement. Finally, in cases where the student was only short one or two credits 
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for promotion to the tenth grade, the school might (provisionally) list the student as a tenth- 
grader, and roster the student into both tenth-grade courses and those ninth-grade courses neces- 
sary for promotion; once the student had passed the ninth-grade courses formally, the school 
would add the student to the tenth grade. Since these flexible arrangements frequently seemed 
to work in the best interests of students, schools preferred to use them rather than creating dis- 
tinct sections and programs for repeating ninth-graders. 

• Major changes in leadership at the district level led to reduced interest 
and support for the Talent Development model. 

The evolving and unplanned nature of Talent Development implementation was gradu- 
ally affected by a number of changes that unfolded at a district-wide level in Philadelphia. The 
most noteworthy were the changes in district management: a state takeover, creation of a School 
Reform Commission, and the swearing-in of a new superintendent in 2002. 31 Within a year of 
his appointment, he brought on a new associate superintendent for secondary schools, who be- 
gan to put his own imprint on high schools in the city. 

Along with a more centralized interest in high schools, there was a newly designed 
standardized curriculum, developed to correspond closely to the educational objectives around 
which the Pennsylvania System of School Assessment (PSSA) test has been organized. The 
initial courses of this new curriculum were introduced during the 2003-2004 school year. Even- 
tually the “core curriculum” is expected to encompass most subjects offered in the district. 

The curriculum’s content is not viewed as inconsistent with the Talent Development ar- 
rangement. However, its structure and timing is built around a traditional (six-period) schedule, 
rather than the extended block scheduling structure, thus posing logistical problems for teachers 
in Talent Development schools. During the 2003-2004 school year, Talent Development liter- 
acy coaches had to make considerable adjustments to the timetable and instructional sequence 
for the district’ s core course, set to be delivered over two semesters, so that it could be success- 
fully delivered in the one-semester, 90-minute block found in a Talent Development school. 

The district also has begun a policy of reducing the size and increasing the number of 
high schools within the city. As a result, some of the “feeder” middle schools that previously 
supplied Talent Development high schools with incoming ninth-graders changed status. Rather 
than moving ninth-graders on, they retained them, and, over several years, the feeder schools 
would themselves become high schools. As a result, the predictable sources of students for the 
Talent Development schools changed, as did the sizes of the incoming classes. 



3 'Between the departure of the district superintendent with whom CRESPAR first made contact and the 
current superintendent’s appointment in 2002, the district has also had two temporary leaders. 
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Perhaps the most striking change to emerge was the district’ s intensive focus on state 
testing and the Adequate Yearly Progress designation it leads to for individual high schools 
that succeed in meeting requirements. 32 One manifestation of this has been the mandatory use 
throughout the district’s high schools of a set of “drill and practice” materials geared to the 
state test. Talent Development schools for the most part adapted them into their daily sched- 
ules as well as they could, although they were at times inconsistent with the emphases of the 
CRESPAR curricula and with the extended block schedule found in the Talent Development 
schools. 33 

By the 2003-2004 school year, it was clear that the somewhat informal arrangements 
with the high schools that Talent Development had previously enjoyed would be constrained. 
The district was far less willing than in previous years to provide fiscal support (through the 
local intermediary) for an initiative such as Talent Development, the tenets of which were not 
wholly consistent with the directions the district seemed likely to follow. In addition, at the dis- 
trict level, there was concern both about the incremental school cost entailed in the extended 
block schedule and, more specifically, about the latitude that the local intermediary and 
CRESPAR had previously been afforded in pooling federal funds that were technically ear- 
marked for individual schools. In summer of 2004, the district took further steps to signal its 
intended changes in direction. 34 



The Analytic Approach and Data Sources of the Impact Study 

In order to determine the net effect of Talent Development, it is necessary to compare 
the experiences of a group of students who were exposed to the model with a truly comparable 
group of students who were not. The ideal research situation would provide an absolutely reli- 
able estimate of the student performance levels that would have been observed in the absence of 
the intervention (that is, a counterfactual) and a comparison of this estimate with actual student 
performance. Random assignment is the most reliable basis from which to construct estimates 
of the counterfactual, but, in this evaluation, it was not possible to randomly assign schools or 



32 Given the low-performing nature of the schools to begin with, it is not surprising that five of the seven 
Talent Development schools were in one or more of the “deficient'’ categories defined by the Commonwealth 
of Pennsylvania in its No Child Left Behind standards. 

33 This represented another claim on the time of Talent Development staff in the schools. 

34 Individual Talent Development schools, somewhat with the encouragement of the district, increased di- 
rect control of the grant funds that were to be spent in their schools; in September 2004, the district insisted that 
all of Talent Development’s Twilight Academy programs be shut down. 



33 




students. The analytic approach that is instead used represents an attempt to construct the best 
counterf actual possible in order to estimate the true impact of Talent Development. 

In this report, impacts are measured using a comparative interrupted time series design 
(see Box l ). j5 This analytic approach combines the use of the interrupted time series analytic 
strategy with the use of the comparison schools analytic strategy to build on the strengths of each 
approach and to address the potential limitations of both. The comparative interrupted time series 
design compares changes or “deviations” from the historical patterns for the Talent Development 
high schools with deviations from the historical patterns for similar non-Talent Development high 
schools during the same period. Thus, impacts are defined as differences between Talent Devel- 
opment and non-Talent Development high schools deviations from historical patterns in student 
outcomes. When combined with regression analysis to control for differences caused by individ- 
ual student background characteristics and prior school experiences, the approach seeks to isolate 
Talent Development's unique impact on student engagement and performance. 

The Interrupted Time Series Methodology 

The interrupted time series component of the analytic strategy assesses the extent to 
which measures of engagement and performance for students in Talent Development high 
schools differ from the engagement and performance for similar students in the same schools 
prior to Talent Development implementation. This provides an indication of whether the par- 
ticipating high schools experienced a deviation from their historical patterns in student out- 
comes that was coincident with the introduction of Talent Development (the “interruption” in 
the interrupted time series design). The projection of each school's recent history into the period 
of Talent Development implementation acts as the counterfactual. This is a particularly good 
counterfactual because, in the absence of the reform, many aspects of the school would be ex- 
pected to stay the same (for example, students, faculty, school culture, neighborhood, and 
physical plant). The use of a historical pattern as the counterfactual has the potential to control 
for both measurable and unmeasurable characteristics of a school. 



35 A detailed description of the analytic approach is available online in the preliminary report’s Technical 
Resources (Unit 1 : Analytic Appendix). See www.mdrc.org/publications/388/techresources.pdf. 
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Box 1 



The Three Steps for Estimating Impacts 
with a Comparative Interrupted Time Series Research Design 

• Step 1: Estimating deviations from baseline in Talent Development schools. 

For each outcome under study, in each Talent Development school, the outcome 
level is compared with the pattern in the same school before it implemented the re- 
form (this is referred to in the report as the “baseline average”). 

• Step 2: Estimating deviations from baseline in non-Talent Development 
comparison schools. For each Talent Development school, the outcome levels in 
a group of comparison schools — a set of schools in the same district with charac- 
teristics similar to those of the Talent Development school — are compared with 
the baseline averages in these schools before the Talent Development school im- 
plemented the reform. 

• Step 3: Estimating the impact of Talent Development. Differences between the 
deviations from the baseline averages in the Talent Development schools and the 
deviations from the baseline averages in the non-Talent Development comparison 
schools are used to estimate the reform’s impact. 



However, the deviation from the baseline alone may not necessarily reflect the impact 
of Talent Development. Similar deviations from historical patterns could have been caused by 
district- wide policies or other interventions that occurred at about the same time as Talent De- 
velopment implementation. For example, while Talent Development was being scaled up, the 
district changed course requirements for grade-level promotion. Such a change may cause posi- 
tive deviations from baseline averages in course credits that were earned at schools in the dis- 
trict. An interrupted time series design alone would capture this improvement and ascribe it to 
Talent Development, but in reality Talent Development may have caused some, all, or none of 
this change in credits earned. In order to sort out what part of the deviation from baseline is 
caused by Talent Development, this study looks at similar high schools in the same district. 

The Comparison Schools Methodology 

The use of non-Talent Development comparison schools helps to account for other fac- 
tors in the broader school district that may influence school functioning and student engagement 
and performance. Each Talent Development school is matched with a set of non-Talent Devel- 
opment comparison schools that are similar on several dimensions. All Talent Development and 
comparison schools are nonselective, comprehensive high schools in a single school district. 
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The schools are matched on racial/ethnic composition, promotion rates of ninth-grade students, 
and the similarity of average test scores and attendance rates. In general, analysis indicates that 
the non-Talent Development comparison schools are similar to the Talent Development schools 
in terms of race/ethnicity, prior test scores, attendance rates, and promotion rates over the years 
leading up to Talent Development implementation. For most student outcomes, however, Talent 
Development schools have slightly lower baseline averages than their comparison schools. In 
general, Talent Development works with the lowest performing schools in the district, so any 
set of comparison schools is likely to be relatively higher performing when compared to the 
Talent Development schools. 

Measures over time of student achievement, course credit attainment, and attendance at 
the comparison schools are another means of estimating what might have been observed in Tal- 
ent Development schools in the absence of the intervention; that is, they can be part of a good 
counterfactual. But differences between the Talent Development and comparison schools do not 
necessarily reflect only the impact of Talent Development. Some differences could be artifacts 
of differences in the prior trends in student engagement and performance. For example, test 
scores for students in Talent Development schools were actually lower than test scores for stu- 
dents in non-Talent Development schools, and they improved only marginally after Talent De- 
velopment began. Suppose, at the same time, test scores for students attending similar schools 
in the district were actually declining over the same period. In this instance, Talent Develop- 
ment may have had a positive impact by preventing test scores from dropping rather than by 
improving the overall average. Such a pattern could be observed only by comparing an inter- 
rupted time series for both Talent Development and non-Talent Development comparison 
schools — that is, by comparing changes over time in Talent Development schools with 
changes over time in non-Talent Development comparison schools. 

Controlling for Changes in School Composition 

This analysis takes into account the fact that Talent Development schools (and non- 
Talent Development comparison schools) may experience a change in the composition of their 
student populations. For example, neighborhoods may undergo demographic changes or 
changes in geographic boundaries or rules governing school assignment. More important, Tal- 
ent Development may cause a change in the student population by, for example, increasing at- 
tendance and reducing mobility and perhaps keeping lower-performing students in school 
longer. In order to help account for systematic changes in the characteristics of student groups 
over time, the analysis incorporates individual student characteristics into the analytic model. 
Specifically, the analytic model includes controls for race/ethnicity, seventh-grade test scores in 
reading and mathematics, and whether students have repeated a prior grade. 
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Analytic Assumptions 

The goal of the comparative interrupted time series approach is to make the causal in- 
ference that Talent Development produced the observed changes, if any, in student outcomes. In 
other words, the analytic approach attempts to distill the impact of Talent Development from 
other factors that may affect student outcomes. In order to make this causal inference, several 
assumptions are made. Box 2 outlines these assumptions, and this section explores the assump- 
tions and provides some context for their validity. 

First, the causal inference is based, in part, on the assumption that the projected baseline 
average for an outcome is a reliable indicator of a given school’s future performance in the ab- 
sence of an intervention like Talent Development or some other event aimed at changing that 
outcome. Year-to-year stability of most outcomes in the baseline period provides some confi- 
dence that this is a reasonable assumption to make for the analysis. Neither Talent Development 
nor non-Talent Development schools displayed substantial positive or negative trends in key 
student outcomes prior to the implementation of the model. 

Second, the causal inference is based, in part, on the assumption that schools with char- 
acteristics similar to Talent Development schools provide a reliable indicator of how student 
outcomes are likely to respond to districtwide policies or events during the Talent Development 
implementation period. Both the comparability of the characteristics of Talent Development and 
non-Talent Development schools and the year-to-year stability of baseline student outcomes in 
both sets of schools suggest that this is a reasonable assumption to make. Analysis indicates that 
Talent Development schools and their comparison schools served similar students with similar 
outcome levels in the baseline period. 

Third, a necessary assumption is that background characteristics of students enrolling in 
the Talent Development and comparison schools do not change over the baseline and follow-up 
period, or that any changes can be controlled for in the statistical model used to estimate im- 
pacts. One competing hypothesis that might explain changes in student outcomes — other than 
the effects of an intervention — is that the composition of the student body changed from the 
baseline to the follow-up period. An intended effect of Talent Development could raise this is- 
sue; the intervention may deter students from dropping out of school. Such students may be less 
able or less motivated and thus lower the school’s average test scores. This analysis assumes 
that variables included in multiple regression impact estimates adequately control for composi- 
tional changes in student characteristics. The analysis accounts for shifts in racial/ethnic compo- 
sition and changes in levels of prior achievement (using seventh-grade test scores), as well as 
whether or not students have repeated a prior grade. There may be changes in other student 
characteristics that correlate with student outcomes — like student motivation — that are not 
included in the regression model because the data are unavailable or cannot be quantified. 



37 




Box 2 



Analytic Assumptions of the 
Comparative Interrupted Time Series Research Design 

• The projected baseline average for an outcome is a reliable indicator of a given 
school’s future performance in the absence of an intervention like Talent Devel- 
opment or an event aimed at changing that outcome. 

• Schools with characteristics similar to Talent Development schools provide a re- 
liable indicator of how student outcomes arc likely to respond to districtwide 
policies or events during the Talent Development implementation period. 

• Background characteristics of students enrolled before Talent Development im- 
plementation arc the same as those of students enrolled in Talent Development 
and comparison schools during implementation of the model (or statistical con- 
trols adequately account for such differences). 

• The factors leading schools to decide to become Talent Development schools do 
not themselves affect student outcomes; for example, Talent Development does 
not attract schools that are already poised to improve student outcomes. 



In short, the analysis in this report represents a very strong application of the compara- 
tive interrupted time series design. Data for three baseline years and up to five follow-up years 
are included in the analysis; impacts are pooled over five Talent Development schools; and be- 
tween two and four comparison schools are matched with each Talent Development school. 
Talent Development and comparison schools are closely matched on demographic and student 
outcome measures. Also, student outcome levels are not so high or so low as to expect that im- 
pact estimates are prone to ceiling or floor effects; student-level covariates were used to control 
for changes in student composition; and the effects are large, consistent, and statistically not 
likely to be due to chance. 

Yet, the findings are not based on an experiment, so even with the strength of this design, 
there may still be alternative explanations or other factors unrelated to Talent Development that 
contribute to the observed differences in student outcomes. For example, the analysis does not 
account for the process by which schools enter into the Talent Development network. Some may 
argue that schools with more entrepreneurial leaders — who are more likely to seek out a reform 
model like Talent Development — may experience improved student outcomes even in the ab- 
sence of the intervention. The analysis is unable to rule out this possibility. Even recognizing the 
limitations of the comparative interrupted time series approach, this methodology offers an unusu- 
ally reliable estimate of the impact of Talent Development in high schools in the district that 
should be interpreted in light of the previous implementation findings. 
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Analysis Sample and Data Sources 

The analysis sample includes students who began ninth-grade in one of five Talent De- 
velopment high schools or six non-Talent Development comparison high schools in the School 
District of Philadelphia. Students in the sample were included on the district’s transcript and at- 
tendance records. The sample excludes students designated as English for Speakers of Other Lan- 
guages (ESOL) or special education, and due to limitations in consistency of the data over time, 
the sample includes only students who attempted at least one course credit in the ninth grade. 

Impact estimates are pooled across the five high schools that are the primary focus of 
this study. By pooling estimates, the analysis has a large enough sample to assess the likelihood 
that a nonzero impact results from chance. In general, the larger the number of schools that ex- 
hibit a nonzero impact, the higher the likelihood that the analysis can detect real changes in stu- 
dent engagement and performance that were produced by Talent Development. Although the 
focus is on results from pooled estimates, results for a smaller subset of schools are also dis- 
cussed in order to explore student outcomes through the fourth year of high school. It should be 
noted, however, that statistical significance, 36 which depends in part on sample size, may be 
achieved with impacts of a smaller magnitude in the estimates pooled over five schools than 
with estimates pooled over two schools. Also, impact estimates for this smaller sample of 
schools may not be representative of the impact of Talent Development in all high schools in 
the district. 

The primary sources of data for the impact analysis are individual students’ school re- 
cords, which were obtained from the district. In general, administrative, attendance-related, and 
course-related information was obtained for all middle and high school students in the district at 
the end of each school year, from 1995-1996 through 2003-2004. Box 3 defines several key 
outcomes included in the analysis. 



36 Statistical significance is a measure of the degree of certainty that some nonzero deviation from the 
baseline average actually occurred. For example, if an impact estimate is statistically significant, then one may 
conclude with some confidence that the program really had an effect. If an impact estimate is not statistically 
significant, then the nonzero estimate is more likely to be the product of chance or random variation in the 
averages that were calculated across the schools and the years under study. Unless otherwise noted, the 
deviations from baseline averages and the Talent Development impacts discussed in this report are statistically 
significant at the 10 percent level or less; that is, there is no more than a 10 percent probability that the 
difference results only from chance or random variation. 
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Attendance 



Box 3 

Definitions of Key Program Outcomes 



• Attendance rate: The total number of days a student was marked as present during a 
school year, divided by the total number of days the student was listed as being enrolled. 

• Chronic absenteeism: When a student had an attendance rate of 80 percent or lower for 
the year. 

• Regular attendance: When a student had an attendance rate of 90 percent or higher for 
the year - . 

Course Credits Earned 

• Total credits earned: A cumulative total of all the credits a student earned over the 
course of the first year of high school, the first two years of high school, the first three 
years of high school, and the first four years of high school. 

• Course credits earned: Indicators of whether a high school student earned course credits 
in selected subject areas, including English, mathematics, science, and algebra. 

• Earned basic academic curriculum credits: A designation indicating that a student 
completed a basic academic curriculum, earning at least 5 credits during the ninth grade 
with 3 of those credits being in mathematics, English, and science. Because this designa- 
tion relied on the district course-credit code, which does not distinguish between elective 
courses and required courses, the designation may include credits that students earned for 
elective courses or for required courses. 

• Earned core graduation credits: A designation indicating that a student completed an 
academic curriculum of at least 23.5 credits, earning at least 4 credits in English, at least 3 
credits in math, at least 3 credits in science, and at least 3 credits in social studies. Be- 
cause this designation relied on the district’s course credit code, which does not distin- 
guish between elective courses and required courses, the designation may include credits 
that students earned for elective courses or for required courses. 

(continued) 
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Box 3 (continued) 



Enrollment Status 

• Promotion and retention: An indicator of whether or not a student was promoted on 
time to the tenth, eleventh, and twelfth grades. By looking ahead one year, it is possible to 
determine if a ninth-grade student is enrolled in the tenth grade or repeating the ninth 
grade by the end of that year. By looking ahead two years, it is possible to determine if a 
ninth-grade student is enrolled in the eleventh grade or repeating either the ninth or tenth 
grades. By looking ahead three years, it is possible to determine if a ninth-grade student is 
enrolled in the twelfth grade or repeating either the ninth, tenth, or eleventh grade. 

• Exited the school system: A designation indicating that a student is no longer in district 
records in subsequent years. Students who exit the school system may have dropped out 
of school, transferred to a public school outside of the district, or transferred to a private 
school. 

Pennsylvania System of School Assessment (PSSA) 

• Pennsylvania System of School Assessment (PSSA) is a criterion-referenced test ad- 
ministered in grades 5, 8, and 1 1, which provides information on student performance 
on skills and content knowledge specified by the state. 

Average scaled score: Indicators of scores received by students on state 
math and reading tests. The scaled score is based on the number correct but 
is transformed to remove the effect of test length and item difficulty. The 
scaled-score metric is anchored to the mean school-level scaled score for a 
base year and that point is arbitrarily labeled 1300. 
ri At below-basic level: An indicator that a student received a score that corre- 
sponds to inadequate academic performance. 

At basic level: An indicator that a student received a score that corresponds 
to marginal academic performance. 

ri At or above proficient level: An indicator that a student received a score that 
corresponds to satisfactory or superior academic performance. 
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Impact Findings 

There are three stories to tell in this final report. First, Talent Development produced 
strong impacts on credits earned, promotion rates, and attendance rates during the first year of 
high school. These effects extend to virtually all cohorts of first-time ninth-graders in the five high 
schools that are the focus of the evaluation. 7 Second, the improvements relative to the baseline for 
cohorts of first-time ninth-grade students in Talent Development high schools continue to outpace 
those seen in comparison school cohorts as they move through high school — earning more total 
credits and earning required credits in math and English. Impacts for nearly all first-time ninth- 
grade student outcomes included in this analysis are positive and statistically significant, meaning 
that the improvement is not likely due to chance. Third, Talent Development had mixed results for 
students who were repeating the ninth grade. 

The initial report in the Talent Development High School evaluation focused on the 
effects of the intervention on ninth-grade students. This final report estimates the impact of 
Talent Development on cohorts of ninth-grade students as they move though high school, 
primarily the first, second, and third year of high school. Table 4 shows that, as of the 2003- 
2004 school year, data are available for three cohorts of students in the baseline period (before 
Talent Development implementation) and up to five cohorts of students in the follow-up pe- 
riod (after Talent Development implementation) in each of five Talent Development high 
schools. The analysis equally weights each school and pools estimates across the schools, so 
that Implementation Year 1 Cohort (Year 1 Cohort, for short) includes students enrolled in the 
ninth grade during the first year of Talent Development implementation in their school, Im- 
plementation Year 2 Cohort (Year 2 Cohort, for short) includes students enrolled in the ninth 
grade during the second year of Talent Development implementation in their school, and the 



37 As discussed in the third section of this report, three other high schools in Philadelphia began implemen- 
tation of Talent Development during the time period included in this study. One high school in the district be- 
gan implementation of Talent Development in 2002-2003 (and subsequently closed), and two high schools in 
the district began Talent Development in the 2003-2004 school year. Data are available only for one cohort of 
ninth-grade students in each of these schools. The present analysis attempts to capture the greatest number of 
cohorts common to the most schools and does not include the three schools that undertook Talent Development 
implementation most recendy. 
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The Talent Development Evaluation 
Table 4 

Implementation Cohorts of Ninth-Grade Students in Five High Schools 



School Year 

1999-2000 2000-2001 



2001-2002 2002-2003 



School C 



I 



Baseline Cohorts 



I 1997-1998, 1998-1999, 1999-2000 



Year 1 Cohort 



Year 2 Cohort 



[ Baseline Cohorts 1 

School A j 1996-1997,1997-1998,1998-1999 


Year 1 Cohort 




Year 2 Cohort 




Year 3 Cohort 




Year 4 Cohort 










j Baseline Cohorts ! 

School B ! 1996-1997,1997-1998,1998-1999 


Year 1 Cohort 




Year 2 Cohort 




Year 3 Cohort 




Year 4 Cohort 



Year 3 Cohort 



School D 


I Baseline Cohorts 1 

1 1998-1999,1999-2000,2000-2001 


Year 1 Cohort 




Year 2 Cohort 










School E 


1 Baseline Cohorts 1 

1 1998-1999,1999-2000,2000-2001 


Year 1 Cohort 




Year 2 Cohort 



2003-2004 



Year 5 Cohort 



Year 5 Cohort 



Year 4 Cohort 



Year 3 Cohort 



Year 3 Cohort 



NOTE: Boxes with dashed edges denote baseline years. Boxes with solid edges denote years of Talent Development implementation. The impact analysis 
estimates impacts for the cohorts of entering students as they move through high school. 









Implementation Year 3 Cohort (Year 3 Cohort, for short) includes students enrolled in the ninth 
grade during the third year of Talent Development implementation in their school. 

For each group of ninth-graders, in both Talent Development and non-Talent Develop- 
ment schools, one-year, two-year, and three-year high school transcripts (grade-level promotion 
and credits earned in math, English, science, social studies, foreign language, and electives) have 
been assembled. Some students were able to progress through high school on time, reaching elev- 
enth grade in three years, while many others were not. However, the analyses are able to gauge the 
progress of each student in accumulating credits in elective courses and courses required for 
grade-level promotion and eventual graduation. Students who no longer appear on school-district 
records after ninth grade are assumed to have earned no credits and to have exited the system, 
whether as dropouts or as transfers out of the district. For some cohorts, data is not available to 
create two- and three-year transcripts because data were collected only through the 2003-2004 
school year. For example, impact estimates for outcomes in the second year of high school are not 
available for the Year 3 Cohort, and impact estimates for the third year of high school are not 
available for the Year 2 and Year 3 Cohorts. 

The findings in this section are presented in two ways. First, summary tables show aver- 
age impacts for multiple outcomes for three cohorts of ninth-grade students. Second, bar graphs 
show the deviations from baseline averages for these cohorts of students in both Talent Develop- 
ment and non-Talent Development schools. Impact estimates — which are featured in the sum- 
mary tables and also indicated in the bar graphs — are defined as the difference between devia- 
tions from baseline for students in Talent Development schools and deviations from baseline for 
students in non-Talent Development schools. 

As context, Table 5 shows the average outcome levels for cohorts of students in the baseline 
period in both Talent Development and non-Talent Development schools. The first two columns 
show outcome levels for first-time ninth-grade students and the last two columns show outcome 
levels for repeating ninth-grade students. As discussed in the previous sections, at the start of imple- 
mentation, Talent Development schools tended to be the lowest-performing schools in the district. 
Therefore, it is not surprising that baseline outcome levels in Talent Development schools are 
slightly lower than baseline outcome levels in non-Talent Development comparison schools. 

The findings are presented in an order that mirrors how one might expect Talent Devel- 
opment to affect student success: changes in attendance patterns, completion of academic courses 
in ninth-grade, grade-level promotion, completion of academic courses over two years of high 
school, and completion of academic courses over three years of high school. Impact findings for 
first-time ninth-grade students are presented first, followed by impact findings for repeating ninth- 
grade students, and findings for a subset of schools for which graduation data are available. Impli- 
cations of these findings are discussed in the next section. 
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The Talent Development Evaluation 
Table 5 



Baseline Outcome Levels for Ninth-Grade Students 
in Talent Development and Non-Talent Development Comparison Schools 





First-Time 


Repeating 




Ninth-Grade Students 


Ninth-Grade Students 




Talent 


Non-Talent 


Talent 


Non-Talent 




Development 


Development 


Development 


Development 


Outcome 


Schools 


Schools 


Schools 


Schools 


At the end of the first/second year of high school 

Attendance 3 (%) 










Attendance rate 

Students with an attendance rate of: 


72.8 


76.6 


56.0 


58.1 


90% or higher 


21.4 


27.3 


4.8 


5.2 


80% or lower 


57.3 


49.5 


87.0 


84.6 


Course credits earned 










Total credits earned in first year 


4.31 


4.62 


4.45 


4.78 


5 or more credits for the year b (%) 


53.3 


57.7 


42.1 


44.9 


Basic academic curriculum 3 (%) 


43.4 


47.6 


25.1 


27.0 


At least 1 English credit for the year (%) 


65.7 


67.9 


51.4 


53.4 


At least 1 math credit for the year (%) 


56.1 


65.0 


41.1 


47.5 


At least 1 algebra credit for the year (%) 

At the end of the second/third year of high school 

Enrollment status d (%) 


33.1 


45.2 


22.0 


26.9 


Enrolled in the 10th grade 


59.0 


60.9 


46.1 


47.9 


Enrolled in the 9th grade 


39.1 


36.9 


46.0 


40.9 


Exited the school system 


1.9 


2.2 


8.0 


11.1 


Course credits earned 










Total credits earned over first two years 
At least 2 credits in English 


8.28 


8.68 


6.59 


6.79 


and at least 2 credits in math (%) 


35.9 


40.7 


18.0 


19.5 


At the end of the third/fourth year of high school 

Enrollment status 3 (%) 










Enrolled in 1 1th grade 


47.1 


49.1 


29.6 


28.3 


Enrolled in 9th or 10th grade 


44.2 


42.1 


38.8 


37.8 


Exited the school system 


8.7 


8.8 


31.7 


34.0 


Course credits earned 










Total credits earned over first three years 
At least 3 credits in English 


11.9 


12.4 


8.3 


8.5 


and at least 3 credits in math (%) 


30.7 


33.6 


16.0 


16.1 


At least 17.5 total credits f (%) 


33.8 


34.9 


18.8 


17.7 



(continued) 
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Table 5 (continued) 





First-Time 

Ninth-Grade Students 


Repeating 

Ninth-Grade Students 


Outcome 


Talent 

Development 

Schools 


Non-Talent 

Development 

Schools 


Talent 

Development 

Schools 


Non-Talent 

Development 

Schools 


llth-grade PSSA test score 8 


Took the test on time (%) 


37.1 


40.0 


NA 


NA 


For test-takers: 
Math 


Average scaled score 


1,063.8 


1,069.7 


NA 


NA 


At below-basic level (%) 


85.9 


83.5 


NA 


NA 


At basic level (%) 


11.5 


12.4 


NA 


NA 


At or above proficient level (%) 


2.6 


4.2 


NA 


NA 


Reading 


Average scaled score 


1,036.5 


1,045.2 


NA 


NA 


At below-basic level (%) 


75.7 


74.0 


NA 


NA 


At basic level (%) 


18.5 


19.1 


NA 


NA 


At or above proficient level (%) 


5.9 


6.9 


NA 


NA 



SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 



NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent 
Development high schools. Students in the sample were included on the district's transcript and attendance records. The 
sample excludes students who did not attempt at least one credit during a given school year. First-time 9th-grade 
students were defined as students whose records indicate that they were in the 9th grade in the year under study and in 
the 8th grade in the previous year's administrative data file. Repeating 9th-grade students were defined as students 
whose records indicate that they were in the 9th grade in the year under study and were also in the 9th grade in the 
previous year's administrative data file. 

The levels presented are averages taken over the three years preceeding the implementation of Talent Development 
in each Talent Development school and the same three years for the non-Talent Development comparison schools. 

■“Attendance rates were calculated for each student by dividing the number of days the student was present by the total 
number of days the student was enrolled in a given school year. 

b Until the 1998-1999 school year, 9th-grade students in the district were required to earn four course credits in order 
to be promoted. Beginning in the 1998-1999 school year, minimum requirements for promotion included earning at 
least five credits during the 9th grade, with three of those credits awarded for completing one required course in 
mathematics, one in English, and one in science. In recent years, the distributive requirement has been dropped. 

c “Basic academic curriculum” is a designation indicating that a student completed a basic academic curriculum, 
earning at least five credits during the 9th grade, with three of those credits being in mathematics, English, and science. 
Because this designation relied on the district’s course-credit code, which does not distinguish between elective courses 
and required courses, the designation may include credits that students earned for elective courses or for required 
courses. Some elective courses did not meet the district’s new 1998-1999 promotion requirements, but all of them 
counted toward total credits earned. 

“For the purposes of this analysis, 9th-grade students were considered to have been promoted to 10th grade if they 
were listed as lOth-graders in the next year's administrative data file. 

“For the purposes of this analysis, 9th-grade students were considered to have been promoted to 1 1th grade if they 
were listed as 1 lth-graders in the administrative data file two years after the current year. 

f Students are required to earn 17.5 credits in order to be promoted to the 12th grade. 

Pennsylvania System of School Assessment (PSSA) is a criterion-referenced test administered in grades 5, 8, and 
11, which provides information on student performance on skills and content knowledge specified by the state. 
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Impacts for First-Time Ninth-Grade Students 

First-time ninth-graders are students whose records indicate that they were in the ninth 
grade during the spring of the year under study and in the eighth grade during the previous 
spring. 38 First-time ninth-graders made up approximately two-thirds of the ninth-grade class at 
each school included in the study, and they are the primary focus of Talent Development’s 
Ninth Grade Success Academy. At the schools in the district, first-time ninth-graders received 
the most intense treatment of Talent Development, and there was the greatest “service differ- 
ence” for these students between the Talent Development and non-Talent Development schools. 
As would thus be expected, Talent Development seems to have had the greatest impact on this 
group. 



The findings below show Talent Development's impact on student outcome for the first 
three years of high school for each cohort of first-time ninth-grade students. Only about sixty 
percent of these students progress to tenth grade in two years, and about half progress to elev- 
enth grade in three years. So, the first-time ninth-grade cohorts include students who reach elev- 
enth grade in three years, as well as students who repeat a grade in the second or third year of 
high school. 

Impacts During the First Year of High School 

Table 5 shows that for cohorts of first-time ninth-grade students in the baseline period, 
the average attendance rate in Talent Development schools was 73 percent and the average rate 
in non-Talent Development comparison schools was 77 percent. The table also shows that less 
than a quarter of students in Talent Development schools regularly attended school as indicated 
by attendance rates of 90 percent or better. More than half were chronic absentees with atten- 
dance rates of 80 percent or lower, missing a total of more than seven weeks of school or an 
average of one day each week. 

• Talent Development improved the attendance rate for first-time ninth- 
grade students. 

Figure 2 shows the deviations from the baseline attendance rates for three cohorts of 
first- time ninth-grade students in Talent Development and non-Talent Development schools 
during the follow-up period. The black bars show deviations for Talent Development schools 



"in this analysis, ninth-grade students new to the school district were also included in the sample of first- 
time ninth-grade students. Also, first- time ninth-grade students who attend Twilight Academy are included in 
this sample. 
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The Talent Development Evaluation 
Figure 2 

Impacts on Attendance Rates 
for First-Time Ninth-Grade Students 
in the First Year of High School 



20 i 



■ Talent Development Schools 
□ Non-Talent Development Schools 



15 H 



s 



2 o 



10 



5 I 



Impact = 2.8* 




Impact = 6.7*** 



Impact = 5.9** 




Year 1 Cohort Year 2 Cohort Year 3 Cohort 

Baseline averages: Talent Development schools = 72.8; Non-Talent Development schools = 76.6 



SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent 
Development high schools. Students in the sample were included on the district's transcript and attendance 
records. The sample excludes students who did not attempt at least one credit during a given school year. First- 
time 9th-grade students were defined as students whose records indicate that they were in the 9th grade in the 
year under study and in the 8th grade in the previous year's administrative data file. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars 
represent the deviations from baseline of the non-Talent Development comparison schools. The deviations were 
calculated as the change in outcome level from the three-year pre-implementation baseline average. 

The impact at follow-up was calculated as the difference in deviations from the baseline average between 
Talent Development schools and non-Talent Development schools. Impacts are pooled across five clusters of 
Talent Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development 
implementation. (The first calendar year of implementation varied by school cluster.) Year 2 Cohort includes 
students who began 9th grade during the second year of Talent Development implementation. Year 3 Cohort 
includes students who began ninth grade during the third year of Talent Development implementation. 

A two-tailed t-test was applied to the impacts. Standard errors and statistical significance levels are adjusted to 
account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 
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and the white bars show deviations for non-Talent Development comparison schools. For ex- 
ample, the average attendance rate of Talent Development students in the Year 1 Cohort im- 
proved by nearly 3 percentage points, while the average attendance rate of non-Talent Devel- 
opment students in the Year 1 Cohort improved by about one-tenth of one percentage point. The 
estimated impact of Talent Development is the difference between the deviation from baseline 
in Talent Development schools and the deviation from baseline in non-Talent Development 
schools. In this case, the estimated impact of Talent Development on attendance rates is 2.8 per- 
centage points. Figure 2 also shows that first-year attendance rates improved for Talent Devel- 
opment students in the Year 2 and Year 3 Cohorts, while average attendance rates declined in 
non-Talent Development schools in these cohorts. 

Across cohorts, Talent Development improved attendance rates by an average of 5 per- 
centage points, which correspond to students attending nearly two more weeks of school each 
year. Further, Table 6, which summarizes impacts for several outcomes, shows that Talent De- 
velopment increased, by an average of nearly 8 percentage points, the percentage of students 
with attendance rates of 90 percent or better and decreased, by an average of 1 1 percentage 
points, the percentage of students with attendance rates of 80 or lower. 

• Talent Development increased the total number of credits earned by 
first-time ninth-grade students. 

For groups of students in the baseline period in both Talent Development and non- 
Talent Development schools, the average number of total credits earned during the first year of 
high school was 4.5, which is just below the total credits required for promotion to tenth 
grade. 39 In the follow-up period, the average number of credits earned by Talent Development 
students in the Year 1 Cohort was 0.91 credits more than the baseline average, while the aver- 
age number of credits earned by students in the comparison school cohort was 0.22 credits more 
than the baseline average. Thus, the impact of Talent Development is an increase of 0.69 cred- 
its, as shown in Table 6. There was a similar pattern of improvement for students in the Year 2 
and Year 3 Cohorts. Across cohorts, Talent Development increased the total credits earned by 
about two-thirds of a credit, which represents about one-quarter of the student-level standard 
deviation for this outcome, and more importantly, raises the average level of this outcome be- 
yond the total number of credits required for promotion. As noted earlier, the extended block 
schedule used in Talent Development schools provides the structure for students to attempt 
more credits per year than a traditional schedule. Talent Development makes use of this struc- 
ture by scheduling two math and two English courses for ninth-grade students. 



39 A minimum of five credits is the current requirement for grade level promotion from ninth to tenth grade. 
During the baseline and follow-up periods, the promotion requirements have changed from a minimum of four 
total credits to a minimum of five total credits. In some years, students were required to complete courses in 
math, English, and science as well as earn the total credit minimum. 
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The Talent Development Evaluation 
Table 6 



Impacts on Outcomes for First-Time Ninth-Grade Students 







Impact at Follow-Up 






Year 1 


Year 2 


Year 3 


Cross-Cohort 


Outcome 


Cohort 


Cohort 


Cohort 


Average 


At the end of the first year of high school 










Attendance (%) a 










Attendance rate 


2.8 * 


5 7 *** 


g 9 *** 


5 1 *** 


Impact effect size 

Students with an attendance rate of: 


0.09 * 


0.22 *** 


0.20 *** 


Q i7 *** 


90% or higher 


4.9 


8.8 ** 


9 2 *** 


7 £ *** 


80% or lower 


-5.5 


-13 4 *** 


_14 i *** 


- 1 1.0 *** 


Course credits earned 










Total credits earned 


0.69 * 


0.58 


0.74 ** 


0.67 *** 


Impact effect size 


0.26 * 


0.22 


0.28 ** 


0.25 *** 


5 or more credits for the year 6 (%) 


7.5 ** 


5.0 


8.6 ** 


7 Q *** 


Basic academic curriculunT (%) 


97 ** 


6.5 


8.2 ** 


g 2 *** 


At least 1 English credit for the year (%) 


9 4 ** 


79 ** 


8.5 ** 


g ^ *** 


At least 1 math credit for the year (%) 


11.5 ** 


10.8 ** 


12 4 *** 


11 ^ *** 


At least 1 algebra credit for the year (%) 


17 j ** 


29.6 *** 


26.8 *** 


24 5 *** 


At the end of the second year of high school 










Enrollment status ' 1 (%) 










Enrolled in the 10th grade 


8.5 ** 


7.4 * 




g Q *** 


Enrolled in the 9th grade 


-9.5 ** 


-7.9 * 




_g 7 *** 


Exited the school system 


0.9 


0.5 




0.7 


Course credits earned 










Total credits earned over first two years 


0.93 ** 


0.76 * 




0.85 *** 


Impact effect size 
At least 2 credits in English 


0.18 ** 


0.15 * 




0.16 *** 


and at least 2 credits in math (%) 


15 4 *** 


IQ 9 *** 




13 7 *** 


At the end of the third year of high school 

Enrollment status 6 (%) 










Enrolled in the 1 1th grade 


6.5 * 






6.5 * 


Enrolled in the 9th or 10th grade 


-7.3 * 






-7.3 * 


Exited the school system 


0.8 






0.8 


Course credits earned 










Total credits earned over first three years 


0.92 * 






0.92 * 


Impact effect size 
At least 3 credits in English 


0.12 * 






0.12 * 


and at least 3 credits in math (%) 


10.5 *** 






10.5 *** 


At least 17.5 total credits 1 (%) 


1.9 






1.9 



(continued) 
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Table 6 (continued) 



included earning at least five credits during the 9th grade, with three of those credits awarded 

for completing one required course in mathematics, one in English, and one in science. In recent years, the 

distributive requirement has been dropped. 

c “Basic academic curriculum” is a designation indicating that a student completed a basic academic curriculum, 
earning at least five credits during the 9th grade, with three of those credits being in mathematics, English, and 
science. Because this designation relied on the district's course-credit code, which does not distinguish between 
elective courses and required courses, the designation may include credits that students earned for elective 
courses or for required courses. Some elective courses did not meet the district’s new 1998-1999 promotion 
requirements, but all of them counted toward total credits earned. 

d For the purposes of this analysis, 9th-grade students were considered to have been promoted to 10th grade if 
they were listed as lOth-graders in the next year's administrative data file. 

Tor the purposes of this analysis, 9th-grade students were considered to have been promoted to 1 1th if they 
were listed as 1 lth-graders in the administrative data file two years after the current year. 

'Students are required to earn 17.5 credits in order to be promoted to the 12th grade. 

Pennsylvania System of School Assessment (PSSA) is a criterion-referenced test administered in grades 5, 8, 
and 11, which provides information on student performance on skills and content knowledge specified by the 
state. 



The district changed its requirements for grade-level promotion over the years. During 
most of the baseline periods, ninth-grade students were required to earn four course credits in or- 
der to be promoted to the tenth grade. Early in the follow-up period, students were required to earn 
at least five credits, and, in some years, to also meet distributive requirements in math, English, 
and science. Two outcomes in Table 6 correspond to the most recent promotion requirements for 
ninth-grade students (five or more credits for the year and basic academic curriculum). In each 
case, Talent Development increased the percentage of students meeting the promotion require- 
ments, and this improvement outpaced that at the comparison schools. Across cohorts, the average 
impact of Talent Development was an increase of about 7 to 8 percentage points in the percent of 
first-time ninth-grade students earning credits required for promotion to tenth grade. 

For example, the bottom of Figure 3 shows that in the baseline period, fewer than half 
of first-time ninth-grade students in Talent Development and non-Talent Development schools 
completed a basic academic curriculum, earning at least five credits during the year, with three 
of those credits being in math, English and science. 40 While average outcomes for cohorts of 



40 “Basic academic curriculum” is a designation indicating that a student completed a basic academic cur- 
riculum, earning at least five credits during the ninth grade, with three of those credits being in mathematics, 
English, and science. Because this designation relied on the district’s course credit code, which does not distin- 
guish between elective courses and required courses, the designation may include credits that students earned 
for elective courses or for required courses. Some elective courses did not meet distributive requirements for 
promotion and graduation, but all of them counted toward total credits earned. 
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TheTalent Development Evaluation 
Figure 3 

Impacts on the Percentage of Students Earning Basic Academic Curriculum 
for First-Time Ninth Grade Students 
in the First Year of High School 



■ Talent Development Schools 
□ Non-Talent Development Schools 




Year 3 Cohort 



Baseline averages: Talent Development schools = 43.4; Non-Talent Development schools = 47.6 



SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent Development 
high schools. Students in the sample were included on the district's transcript and attendance records. The sample excludes 
students who did not attempt at least one credit during a given school year. First-time 9th-grade students were defined as 
students whose records indicate that they were in the 9th grade in the year under study and in the 8th grade in the previous 
year's administrative data file. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars represent the 
deviations from baseline of the non-Talent Development comparison schools. The deviations were calculated as the change 
in outcome level from the three-year pre-implementation baseline average. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development implementation. (The 
first calendar year of implementation varied by school cluster.) Year 2 Cohort includes students who began 9th grade 
during the second year of Talent Development implementation. Year 3 Cohort includes students who began 9th grade 
during the third year of Talent Development implementation. 

A two-tailed t-test was applied to the impacts. Standard errors and statistical significance levels are adjusted to account 
for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

"Basic academic curriculum" is a designation indicating that a student completed a basic academic curriculum, earning 
at least five credits during the 9th grade, with three of those credits being in mathematics, English, and science. Because 
this designation relied on the district’ s course-credit code, which does not distinguish between elective courses and 
required courses, the designation may include credits that students earned for elective courses or for required courses. 

Some elective courses did not meet the district’s new 1998-1999 promotion requirements, but all of them counted toward 
total credits earned. 
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students in both sets of schools improved in the follow-up period, gains by students in Talent 
Development schools consistently outpaced gains made by students in comparison schools. In 
the Year 1 Cohort, the percentage of students in Talent Development who earned these credits 
increased by 13 percentage points above the baseline average, while the increase was only 3 
percentage points in non-Talent Development schools. Therefore, the estimated impact of Tal- 
ent Development is nearly 10 percentage points. 

• Talent Development produced substantial gains in academic course 
credits earned by first-time ninth-grade students. The impact was espe- 
cially large for the percentage of students earning a credit in algebra. 

Because of the double-dose courses in English and math in Talent Development 
schools, ninth-grade students in Talent Development schools have more opportunities to earn 
academic course credits in the first year of high school. In Talent Development schools, first- 
semester catch-up courses, which count as elective courses, are designed to prepare students for 
English 1 and algebra, which are required for graduation. It is difficult to determine Talent De- 
velopment’s impact on the completion of English 1 because all English credits are similarly 
coded in the dataset. Impact estimates discussed below include both elective and required 
courses in English. However, completion of an algebra credit is more clearly discerned from the 
available data and, thus, discussed specifically. 

Talent Development had a significant impact on the percentage of first-time ninth-grade 
students earning one or more credits in English and algebra. In the baseline period, about 65 per- 
cent of first-time ninth-grade students earned one or more credits in English. This percentage in- 
creased by 10, 9, and 14 percentage points for successive cohorts of ninth-grade students in Talent 
Development schools. At the same time, this percentage increased by only 1,1, and 5 percentage 
points in successive cohorts of comparison schools. As Table 6 illustrates, this resulted in impacts 
of between 8 and 9 percentage points for each cohort of first-time ninth-grade students. 

The estimated impact of Talent Development was greater for math credit outcomes, 
particularly algebra. In the baseline period, only 33 percent of students in Talent Development 
schools and 45 percent of students in non-Talent Development schools earned one or more 
credits in algebra. This percentage increased by an average of 28 percentage points for cohorts 
of students in Talent Development schools. At the same time, this percentage increased by an 
average of 4 percentage points in non-Talent Development schools. Resulting impacts ranged 
between 17 and 30 percentage points. These impacts represent a near doubling of the percentage 
of students who earned a credit in algebra in Talent Development schools — with levels of 
more than 60 percent in the follow-up period. 
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Impacts Beyond the First Year of High School 

Looking ahead, for each ninth-grade student, enrollment data indicate whether or not 
the student progresses to the next grade, was retained, or left the district. Also, there is a cumu- 
lative record of the total credits earned over two and three years of high school. If students are 
not included in subsequent school record datasets provided by the district, the analysis was un- 
able to add to their cumulative record but did not drop them from the sample. For these stu- 
dents, their two- and three-year cumulative records include only credits earned in the ninth- 
grade. Effectively, students earn zero credits during years they are not in the school system. 
However, since there has been little change in the percentage of students leaving the district, 41 
the following findings represent an estimate of Talent Development’s impact on student pro- 
gress through high school that does not appear to be driven by differential dropout rates in Tal- 
ent Development and non-Talent Development schools. 

• Talent Development improved the overall promotion rate to the tenth 
grade for first-time ninth-grade students. 

Promotion to the tenth grade is defined in this analysis as the percentage of first-time 
ninth-graders who reached the tenth grade by the end of the following school year. It thus in- 
cludes students promoted on-time, as well as students who may have been promoted mid- 
year. 42 In the baseline period, the overall rate of promotion from ninth to tenth grade was about 
60 percent for groups of students in both Talent Development and non-Talent Development 
schools. Figure 4 shows that in the follow-up period, the percentage of students promoted to 
tenth grade improved by 7 percentage points for Talent Development students in the Year 1 
Cohort and 10 percentage points for Talent Development students in the Year 2 Cohort. At the 
same time, the promotion rate fell by 1 percentage point for students in comparison schools in 
the Year 1 Cohort, and rose by 3 percentage points for students in comparison schools in the 
Year 2 Cohort. Therefore, Talent Development improved the overall promotion rate from ninth 



4l There is no significant change in the percentage of first-time ninth-grade students who do not appeal' on 
the school districts’ records after their first and second year of high school. Across the baseline and follow-up 
years, about two percent of first-time ninth-grade students in both Talent Development and non-Talent Devel- 
opment schools exited the school system after their first year of high school. Across the baseline and follow-up 
years, about 9 percent of first-time ninth-grade sUidents in both Talent Development and non-Talent Develop- 
ment schools exited the school system after their second year of high school. This finding includes only ninth- 
grade students in the analysis sample. The sample excludes student classified as SPED or ESOL and sUidents 
who did not attempt at least one credit during the school year. 

42 Since data were obtained from the district at the end of each school year, it is not possible to determine 
direcdy which students achieved tenth-grade status on time and which were promoted midyear. 
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The Talent Development Evaluation 
Figure 4 

Impacts on the Percentage of Students Promoted to Tenth Grade 
for First-Time Ninth-Grade Students 
at the End of the Second Year of High School 
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Year 1 Cohort 



■ Talent Development Schools 
□ Non-Talent Development Schools 



Impact =7.4* 




Year 2 Cohort 



Baseline averages: Talent Development schools = 59.0; Non-Talent Development schools = 60.9 



SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent Development 
high schools. Students in the sample were included on the district's transcript and attendance records. The sample excludes 
students who did not attempt at least one credit during a given school year. Repeating 9th-grade students were defined as 
students whose records indicate that they were in the 9th grade in the year under study and were also in the 9th grade in the 
previous year's administrative data file. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars represent the 
deviations from baseline of the non-Talent Development comparison schools. The deviations were calculated as the change 
in outcome level from the three-year pre-implementation baseline average. 

The impact at follow-up was calculated as the difference in deviations from the baseline average between Talent 
Development schools and non-Talent Development schools. (Impacts are pooled across five clusters of Talent 
Development schools and their matched comparison schools.) 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development implementation. (The 
first calendar year of implementation varied by school cluster.) Year 2 Cohort includes students who began 9th grade 
during the second year of Talent Development implementation. 

A two-tailed t-test was applied to the impacts. Standard errors and statistical significance levels are adjusted to account 
for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

For the purposes of this analysis, 9th-grade students were considered to have been promoted to 10th grade if they were 
listed as lOth-graders in the next year's administrative data file. 
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to tenth grade by an average of about 8 percentage points across two cohorts of first-time ninth- 
grade students. 43 

• For first-time ninth-grade students in Talent Development schools, the 
intervention increased the total credits earned in the first two years of 
high school and improved overall promotion rates to the eleventh grade. 

Students who began ninth grade for the first time in a Talent Development high school 
earned more total credits by the end of their second year of high school than students in non- 
Talent Development comparison schools. In general, students needed at least 1 1 credits to be 
promoted to the eleventh grade. 44 For cohorts of students in the baseline period, the average 
number of total credits earned during the first two years of high school was about 8.5. In the 
follow-up period, the Talent Development students in the Year 1 Cohort earned 1.2 more credits 
than cohorts of students in the baseline period, while comparison school students earn 0.3 more 
credits than cohorts of students in the baseline period. The estimated impact of Talent Devel- 
opment was nearly one full credit (0.93 credits). The Talent Development impact was three- 
quarters of a credit (0.76 credits) for students in the Year 2 Cohort 45 Though neither impact is 
large enough to raise the average credits earned above grade-level promotion requirements, the 
magnitude of the impact is close to one-fifth of the student-level standard deviation and exceeds 
the estimated impact on total credits during the first year of high school. Therefore, the Talent 
Development impact on credits earned seems to have persisted and grown somewhat during the 
second year of high school, which is consistent with double -dose curricular programs in math 
and English for tenth-grade students in Talent Development schools. 

A greater percentage of students who began ninth grade for the first time in Talent De- 
velopment schools earned two credits in math and two credits in English by the end of their sec- 
ond year of high school. This outcome is particularly important because students must earn four 
English credits and at least three credits in math to graduate from high school. In the baseline 
period, only about 40 percent of students had earned these credits by the end of the second year 
of high school. Figure 5 shows that the percentage of students earning two credits in math and 
two credits in English increased in both Talent Development and non-Talent 



43 Promotion, retention, and exit data for all five schools are not available for the third cohort of ninth- 
grade students. 

44 For students who entered high school during the 1998-1999 school year or later, 11 credits were needed 
for promotion from the tenth to eleventh grade, including two credits in English, one credit in social sUtdies, 
two credits in math, two credits in science, and four elective credits. In recent years, the distributive require- 
ments have been dropped. 

45 At this point in the analysis, it is possible to assemble two-year transcripts for only two analysis groups 
of ninth-grade students in all five school clusters. 
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The Talent Development Evaluation 
Figure 5 

Impacts on the Percentage of Students Earning 
at Least Two Credits in English and at Least Two Credits in Math 
for First-Time Ninth-Grade Students 
at the End of the Second Year of High School 




Year 1 Cohort 



Year 2 Cohort 



Baseline averages: Talent Development schools = 35.9 ; Non-Talent Development schools = 40.7 



SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent Development 
high schools. Students in the sample were included on the district's transcript and attendance records. The sample excludes 
students who did not attempt at least one credit during a given school year. First-time 9th-grade students were defined as 
students whose records indicate that they were in the 9th grade in the year under study and in the 8th grade in the previous 
year's administrative data file. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars represent the 
deviations from baseline of the non-Talent Development comparison schools. The deviations were calculated as the change 
in outcome level from the three-year pre-implementation baseline average. 

The impact at follow-up was calculated as the difference in deviations from the baseline average between Talent 
Development schools and non-Talent Development schools. Impacts are pooled across five clusters of Talent 
Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development implementation. (The 
first calendar year of implementation varied by school cluster.) Y ear 2 Cohort includes students who began 9th grade 
during the second year of Talent Development implementation. 

A two-tailed t-test was applied to the impacts. Standard errors and statistical significance levels are adjusted to account 
for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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Development schools. Across cohorts, Talent Development gains outpaced gains in the com- 
parison school cohorts by 16 and 1 1 percentage points. 

Improvements in course completion during the first two years of high school seem to 
have translated into impacts on rates of promotion to the eleventh grade, which is based on a 
student’s enrollment status at the end of the third year of high school. 46 Though data are only 
available for students in the Year 1 Cohort, the findings show that the percentage of students 
promoted to eleventh grade rose by 6 percentage points in Talent Development schools (from 
47 percent), while this percentage fell by half of one percentage point in comparisons schools 
(from 50 percent). Therefore, Talent Development improved the overall promotion rate to elev- 
enth grade by 6.5 percentage points. 

• For first-time ninth-grade students in Talent Development schools, the 
intervention increased the total credits earned in the first three years of 
high school. 

Currently in the district, students must earn 17.5 credits in order to be promoted to the 
twelfth grade. For cohorts of students in the baseline period, the average number of total credits 
earned during the first three years of high school was about 12. In the follow-up period, Talent 
Development students in the Year 1 Cohort earned 1.3 more credits than students in the baseline 
period, while comparison school students in the Year 1 Cohort earned 0.4 more credits than stu- 
dents in the baseline period. As a result, students who began ninth grade for the first time in a 
Talent Development high school earned more total credits by the end of their third year of high 
school than students in non-Talent Development comparison schools. The estimated impact of 
0.92 is essentially the same as the impact on total credits earned after two years of high school. 
Therefore, the impact on credits earned that at emerged after two years of high schools was sus- 
tained but did not increase in the third year of high school. This is consistent with Talent Devel- 
opment implementation, which does not include curricular components for eleventh grade. 

Talent Development also affected the percentage of first-time ninth-grade students who 
earned three credits in math and three credits in English by the end of their third year of high 
school. In the baseline period, only about a third of students earned these credits in both the Tal- 
ent Development and non-Talent Development schools. Figure 6 shows that for the Year 1 Co- 
hort, the percentage of students earning three credits in math and three credits in English rose by 
15 percentage points (to 46 percent) in Talent Development schools and rose by 5 percentage 
points (to 38 percent) in non-Talent Development schools. While the gain in Talent 



46 Since student records data are collected at the end of each school year, it is not possible to determine if 
sUidents were promoted on time or were promoted mid-year. Therefore, promotion outcomes may be inconsis- 
tent with course-credit outcomes. 
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The Talent Development Evaluation 
Figure 6 

Impacts on the Percentage of Students Earning 
at Least Three Credits in English and at Least Three Credits in Math 
for First-Time Ninth-Grade Students 
at the End of the Third Year of High School 
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Impact = 10.5*** 




Year 1 Cohort 



■ Talent Development Schools 
□ Non-Talent Development Schools 



Baseline averages: Talent Development schools = 30.7; Non-Talent Development schools = 33.6 



SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent Development 
high schools. Students in the sample were included on the district's transcript and attendance records. The sample excludes 
students who did not attempt at least one credit during a given school year. First-time 9th-grade students were defined as 
students whose records indicate that they were in the 9th grade in the year under study and in the 8th grade in the previous 
year's administrative data file. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars represent the 
deviations from baseline of the non-Talent Development comparison schools. The deviations were calculated as the 
change in outcome level from the three-year pre-implementation baseline average. 

The impact at follow-up was calculated as the difference in deviations from the baseline average between Talent 
Development schools and non-Talent Development schools. Impacts are pooled across five clusters of Talent 
Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development implementation. 

(The first calendar year of implementation varied by school cluster.) 

A two-tailed t-test was applied to the impacts. Standard errors and statistical significance levels are adjusted to account 
for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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Development schools outpaced the gain in the comparisons schools by over 10 percentage 
points, this impact is smaller than reported for a similar outcome at the end of the second year of 
high school. Again, the Talent Development impact on course credits earned does not seem to 
grow during the third year of high school. 

• Talent Development produced slight improvements in student perform- 
ance on the state standards assessment in math and produced no sys- 
tematic change in reading scores. 

In recent years, the district has placed greater importance on the Pennsylvania System 
of School Assessment (PSSA) as a measure of student achievement and school success. The 
PSSA is administered to fifth-, eighth- and eleventh-grade students. The eleventh-grade PSSA 
test scores are available for students in the Year 1 Cohort who took the tests by the end of their 
third year of high school. The second page of Table 5 shows that less than 40 percent of first- 
time ninth-grade students took the eleventh-grade state standards assessments on time. Some 
non-test takers have exited the school system, either as dropouts or as transfers. Others have 
been retained in grade and will take the test in another year or were exempt from testing. In any 
case, the percentage of students taking the tests is increasing in both Talent Development and 
non-Talent Development schools. On average, improvement in the percentage of students tak- 
ing the test on time in Talent Development schools has outpaced non-Talent Development 
schools by about 4 percentage points, but these differences are not statistically significant. 

In general, this seems to be the pattern for each of the test score outcomes. The findings 
show improvement from the baseline averages in both Talent Development and non-Talent De- 
velopment schools with few systematic differences between the two groups of schools. One 
exception is the percentage of students scoring at the “below basic” level in math. Since the vast 
majority of students who took the test scored in this category, any improvement should be seen 
as a decrease in the percentage of student in this group. In fact, the percentage of students in the 
below basic category in math decreased by 10 percentage points in Talent Development 
schools, while the percentage decreased by 4 percentage points in non-Talent Development 
comparison schools. The 6 percentage -point difference between these changes, which is the es- 
timated impact of Talent Development, is small but statistically significant. 

These test-score findings in the third year of high school seem somewhat inconsistent 
with substantial improvements in course credits earned and rates of promotion during the first 
two years of high school. One possible explanation is that the composition of students in Talent 
Development schools changed significantly due to the intervention. For example, lower- 
performing students who, without the intervention, may have dropped out of school have stayed 
in school and dampened potential test score gains. However, the findings do not indicate a sig- 
nificant increase in eleventh-grade test-takers or substantial change in the characteristics of test- 
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takers in Talent Development schools. 47 A second explanation is that, in the upper grades, the 
experience of students in Talent Development schools is not that different than the experience 
of students in the baseline period or the experience of students in comparison schools. This ex- 
planation seems more likely given Talent Development’s curricular focus in the ninth and tenth 
grades and given the fact that Career Academies (the primary intervention for upper grades) are 
in place in most high schools in the district. Alternatively, findings for the Year 1 Cohort, which 
includes students who began ninth grade in Talent Development schools during the first year of 
implementation, represent impacts for a relatively immature version of Talent Development in 
the upper grades. 48 

Impacts for Repeating Ninth-Grade Students 

While the Talent Development model aims to reduce the number of students who repeat 
the ninth grade by giving intensive support to first-time ninth-grade students, the ideal version 
of the model specifically targets students entering the ninth grade for a second time. As dis- 
cussed earlier, the model calls for students who fall into this category to have their own team 
within the Ninth Grade Success Academy and to work to earn the course credits necessary for 
midyear promotion. However, in practice, repeaters were not given this high level of specialized 
support. Repeaters may have benefited from other aspects of the Ninth Grade Success Acad- 
emy, including the double block schedule, which may facilitate catching up on credits. In gen- 
eral, Talent Development may have the largest impact on repeating ninth-grade students by re- 
ducing their numbers — that is, by reducing the percentage of first-time ninth-grade student 
who are retained in grade. 

For the purposes of this study, repeating ninth-graders are students whose records indi- 
cate that they were in the ninth grade during the spring of the year under study and were also in 
the ninth grade during the previous spring. They did not earn sufficient credits to be promoted to 
the tenth grade on time nor to be promoted midyear 49 About one-third of the ninth-grade stu- 
dents in the study’s sample meet these criteria and are included in the analysis for repeating 



47 For students who took the eleventh-grade state standards assessment. Talent Development had a small 
but negative impact on eighth-grade test scores and eighth-grade attendance rates. However, the change in the 
characteristics of cohorts of test-takers in the follow-up period as compared to characteristics of test-takers in 
the baseline period and characteristics of test-takers in comparison schools is not statistically significant. 

48 Findings for the Year 2 and Year 3 Cohorts are available for a smaller sample of two schools and are 
discussed at the end of this section. 

49 It is difficult to determine the effectiveness of this intervention for all students who begin a second year 
of ninth grade, because the data only allow for analysis of students who repeated a full year or more. Repeating 
ninth-grade students promoted midyear to the tenth grade are classified as tenth-graders in school records data 
for the year in which they are promoted and are not part of the repeater sample. Data from their first year in 
ninth grade are captured in the ninth-grade sample one year earlier. 
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ninth-grade students. Most of these students were repeating the ninth grade for the first time, 
although some were repeating for the second or even third time. 

• In the pre-Talent Development baseline period, repeating ninth-grade 
students have lower attendance rates, are less likely to earn academic 
credits, and are less likely to be promoted to tenth grade than first-time 
ninth-grade students. 

Table 5 presents the average outcome levels for students in the pre-Talent Development 
baseline period. The first two columns contain averages for first-time ninth-grade students in 
Talent Development and non-Talent Development schools, respectively. The third and fourth 
columns contain averages for repeating ninth-grade students in Talent Development and non- 
Talent Development schools, respectively. There are two important comparisons. First, outcome 
levels in Talent Development schools tend to be lower than outcome levels in comparison 
schools. In general, Talent Development works with the lowest performing schools in the dis- 
trict. Second, outcome levels for repeaters tend to be significantly lower than outcome levels for 
first-time ninth-grade students. For example, the average attendance rate for first-time ninth- 
grade students in Talent Development schools is 73 percent. The attendance rate for repeating 
ninth-grade students is 56 percent. On average, repeaters miss 30 more days of schools (over 6 
weeks) than first-time ninth-grade students. Repeaters are also less likely to earn academic cred- 
its and be promoted to tenth and eleventh grade. 

• Talent Development decreased the percentage of students who were re- 
peating the ninth grade in the last two implementation cohorts. How- 
ever, there was little change in the characteristics of repeating ninth- 
grade students in Talent Development schools. 

As expected, Talent Development did not change the percentage of students in the Year 
1 Cohort who were repeating ninth grade. Upon entering ninth-grade, these students had not 
received any Talent Development treatment, and the percentage of repeaters in the full ninth- 
grade sample did not differ much from the percentage of repeaters in the baseline period or from 
the percentage in comparison schools. As discussed earlier, Talent Development had a positive 
impact on the percentage of ninth-grade students promoted to the tenth grade. Therefore, later 
cohorts of ninth-grade students in Talent Development schools experienced a small decrease in 
the percentage of students repeating the ninth grade. 

Before looking at impacts for repeating ninth-grade students, it is important to assess 
whether or not their characteristics change due to Talent Development’s impact on promotion. In 
fact, in Talent Development schools, there appears to be some change in the average characteris- 
tics of repeating ninth-grade students in the Year 2 Cohort. Based on middle school test scores and 
attendance rates, these repeating ninth-graders are somewhat lower-performing and may be less 
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engaged than repeaters in the baseline period and in comparison schools. For example, in the 
baseline period, the average NCE score on the seventh-grade reading assessment was about 30. In 
Talent Development schools, this average fell by 2 NCE points, while there was almost no change 
in non-Talent Development schools. At the same time, average eighth-grade attendance rates were 
3 percentage points higher in non-Talent Development schools and increased by less than one 
percent in Talent Development schools. Similar changes in composition were not found in the 
Year 3 Cohort. Therefore, the findings discussed below do not seem to be driven by changes in 
the composition of cohorts of repeating ninth-grade students over time. 

Impacts after the First Year of High School 

By definition, repeating ninth-grade students have attended high school for more than 
one year. The attendance outcomes discussed below are based on data from the repeated ninth- 
grade year, which in most cases is the second year of high school. 50 Course credit outcomes are 
cumulative and include data from each student’ s first year in ninth grade, repeated year in ninth 
grade, and subsequent years in high school where applicable. Table 7 provides a summary of 
Talent Development’s impact for students who repeated the ninth grade for a full year or more. 
The first panel of the table shows outcomes for their second year of enrollment in ninth grade. 

• Talent Development improved attendance outcomes for repeating ninth- 
grade students. But, on average, students still miss more than ten weeks 
of school. 

Talent Development appears to have a substantial impact on the attendance rates of re- 
peating students in the Year 2 Cohort, which is associated with the second year of Talent De- 
velopment implementation. These students have experienced Talent Development for two years 
— typically, once as a first-time ninth-grader and once as a ninth-grade repeater. In contrast, 
repeating students in the Year 1 Cohort only experienced Talent Development once. Students in 
the Year 3 Cohort also experienced Talent Development for two years, but Talent Development 
did not have the same impact on attendance rates for this cohort. 

Figure 7 shows that for each cohort, average attendance rates improved as compared to 
baseline means in both Talent Development and non-Talent Development schools; both were 
below sixty percent. However, improvements in Talent Development schools exceed those in 
comparison schools. For the Year 2 Cohort, attendance rates for Talent Development students 
improved by 1 1 percentage points, while attendance rates for non-Talent Developments stu- 
dents improved by 2 percentage points, which resulted in an estimated impact of 9 percentage 
points. Even with this improvement, attendance rates for repeating ninth-grade 



50 A small percentage of repeating ninth-grade students are repeating the grade for a second time. 
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The Talent Development Evaluation 
Table 7 



Impacts on Outcomes for Repeating Ninth-Grade Students 







Impact at Follow-Up 






Year 1 


Year 2 


Year 3 


Cross-Cohort 


Outcome 


Cohort 


Cohort 


Cohort 


Average 


At the end of the second year of high school 

Attendance (%) a 










Attendance rate 


3.0 


g 3 *** 


4.6 


5.6 *** 


Impact effect size 


0.10 


0.31 *** 


0.15 


0.19 *** 


Students with an attendance rate of: 










90% or higher 


0.0 


3.9 * 


0.9 


1.6 


80% or lower 


-1.3 


-9.5 ** 


0.1 


-3.6 


Course credits earned 










Total credits earned 


-0.30 


0.06 


-0.45 


-0.23 


Impact effect size 


- 0.11 


0.02 


-0.17 


-0.09 


5 or more credits for the year b (%) 


-7.3 


-3.0 


- 8.1 


- 6.1 ** 


Basic academic curriculum 1 (%) 


-5.2 


- 1.2 


-5.6 


-4.0 * 


At least 1 English credit for the year (%) 


-1.9 


0.9 


-1.9 


- 1.0 


At least 1 math credit for the year (%) 


-7.4 


0.1 


- 0.6 


- 2.6 


At least 1 algebra credit for the year (%) 


- 8.1 


1.2 


5.1 


- 0.6 


At the end of the third year of high school 










Enrollment status ' 1 (%) 










Enrolled in the 10th grade 


1.9 


-2.3 




- 0.2 


Enrolled in the 9th grade 


-7.7 


- 2.8 




-5.3 


Exited the school system 


5.7 


5.1 




5.4 ** 


Course credits earned 










Total credits earned over first two years 


-0.79 


-0.50 




-0.65 


Impact effect size 
At least 2 credits in English 


-0.15 


- 0.10 




- 0.12 


and at least 2 credits in math (%) 


-4.2 


-1.9 




-3.1 


At the end of the fourth year of high school 

Enrollment status 6 (%) 










Enrolled in the 1 1th grade 


.7 7 ** 






_7 7 ** 


Enrolled in the 9th or 10th grade 


-3.6 






-3.6 


Exited the school system 


11.3 






11.3 


Course credits earned 










Total credits earned over first three years 


-1.10 






-1.10 


Impact effect size 
At least 3 credits in English 


-0.14 






-0.14 


and at least 3 credits in math (%) 


-3.8 






-3.8 


At least 17.5 total credits 5 (%) 


-4.3 






-4.3 



(continued) 
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Table 7 (continued) 

SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent 
Development high schools. Students in the sample were included on the district's transcript and attendance records. 
The sample excludes students who did not attempt at least one credit during a given school year. Repeating 9th- 
grade students were defined as students whose records indicate that they were in the 9th grade in the year under 
study and were also in the 9th grade in the previous year's administrative data file. 

The impacts at follow-up were calculated as the difference in deviations from the baseline average between 
Talent Development schools and non-Talent Development schools. Impacts are pooled across five clusters of 
Talent Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began ninth grade during the first year of Talent Development 
implementation in their school. (The first calendar year of implementation varied by school cluster.) Year 2 Cohort 
includes students who began ninth grade during the second year of Talent Development implementation in their 
school. Year 3 Cohort includes students who began ninth grade during the third year of Talent Development 
implementation in their school. 

The impact effect size was calculated for continuous variables by dividing the impact at follow-up by the standard 
deviation of the outcome for all 9th-grade students in the district's nonselective, comprehensive high schools from 
school years 1996-1997 through 1998-1999. 

Estimates are regression-adjusted using ordinary least squares, controlling for 7th-grade math and reading SAT-9 
test scores, race, and whether the student had repeated a prior grade. 

A two-tailed t-test was applied to the impacts at follow-up. Standard errors and statistical significance levels are 
adjusted to account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

"Attendance rates were calculated for each student by dividing the number of days the student was present by the 
total number of days the student was enrolled in a given school year. 

b Until the 1998-1999 school year, 9th-grade students in the district were required to earn four course credits in 
order to be promoted. Beginning in the 1998-1999 school year, minimum requirements for promotion included 
earning at least five credits during the 9th grade, with three of those credits awarded for completing one required 
course in mathematics, one in English, and one in science. In recent years, the distributive requirement has been 
dropped. 

‘'"Basic academic curriculum” is a designation indicating that a student completed a core academic curriculum, 
earning at least five credits during the 9th grade, with three of those credits being in mathematics, English, and 
science. Because this designation relied on the district’s course-credit code, which does not distinguish between 
elective courses and required courses, the designation may include credits that students earned for elective courses 
or for required courses. Some elective courses did not meet the district’s new 1998-1999 promotion requirements, 
but all of them counted toward total credits earned. 

d For the purposes of this analysis, 9th-grade students were considered to have been promoted to 10th grade if they 
were listed as lOth-graders in the next year's administrative data file. 

Tor the purposes of this analysis, 9th-grade students were considered to have been promoted to 1 1th if they were 
listed as 1 lth-graders in the administrative data file two years after the current year. 

r Students are required to earn 17.5 credits in order to be promoted to the 12th grade. 
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The Talent Development Evaluation 
Figure 7 

Impacts on Attendance Rates 
for Repeating Ninth-Grade Students 
at the End of the Second Year of High School 
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Year 1 Cohort Year 2 Cohort Year 3 Cohort 

-5 J 

Baseline averages: Talent Development schools =56.0; Non-Talent Development schools = 58.1 
SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 



NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent 
Development high schools. Students in the sample were included on the district's transcript and attendance 
records. The sample excludes students who did not attempt at least one credit during a given school year. 
Repeating 9th-grade students were defined as students whose records indicate that they were in the 9th grade in 
the year under study and were also in the 9th grade in the previous year's administrative data file. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars 
represent the deviations from baseline of the non-Talent Development comparison schools. The deviations were 
calculated as the change in outcome level from the three-year pre-implementation baseline average. 

The impact at follow-up was calculated as the difference in deviations from the baseline average between 
Talent Development schools and non-Talent Development schools. Impacts are pooled across five clusters of 
Talent Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development 
implementation. (The first calendar year of implementation varied by school cluster.) Year 2 Cohort includes 
students who began 9th grade during the second year of Talent Development implementation. Year 3 Cohort 
includes students who began 9th grade during the third year of Talent Development implementation. 

A two-tailed t-test was applied to the impacts. Standard errors and statistical significance levels are adjusted to 
account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Attendance rates were calculated for each student by dividing the number of days the student was present by the 
total number of days the student was enrolled in a given school year. 
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students are below 70 percent, which means students miss more than ten weeks of school each 
year. Given such low attendance rates, it seems unlikely that an in-school intervention, like Tal- 
ent Development, would have much impact on course completion and promotion outcomes. 

• Though Talent Development did not produce impacts on credits earned 
or enrollment status, repeating ninth-grade students in both Talent De- 
velopment and non-Talent Development schools earned more academic 
credits by the end of their second and third years of high school than 
students in the baseline period. 

Figure 8 illustrates the trend for most course credit outcomes. In the baseline period, 
fewer than 20 percent of students in Talent Development and non-Talent Development schools 
completed a basic academic curriculum — earning at least five credits with one credit each in 
math, English, and science. In the follow-up period, the percentage of students earning these 
credits increased in both Talent Development and non-Talent Development schools by between 
1 and 10 percentage points. However, progress in Talent Development schools did not exceed 
progress in comparison schools. Figure 9 shows that there was little change in overall promo- 
tion rates to tenth grade for repeating ninth-grade students in both Talent Development and non- 
Talent Development schools. There was no significant difference between the change in promo- 
tion rates in Talent Development schools and the change in comparison schools. 

• For students who repeated the ninth grade in a Talent Development 
school, the intervention produced a negative impact on promotion rates 
to eleventh grade. 

In the baseline period, about one -third of repeating ninth-grade students were in the 
eleventh grade at the end of their fourth year of high school. About one- third had exited the 
school system and another third were retained in grade. Data through the fourth year of high 
school are only available for the first cohort of students. Students in the Year 1 Cohort who can 
be followed through the third year of high school were exposed to Talent Development as re- 
peating ninth-grade student but not as first-time ninth-grade students. For students in the Year 1 
Cohort, the two-year promotion rate from ninth to eleventh grade declined by 6 percentage 
points in Talent Development schools and increased by 2 percentage points in non-Talent De- 
velopment schools. Thus, the impact of Talent Development was an 8 percent point decrease in 
promotion to eleventh grade. 

Over 90 percent of repeating ninth-grade students in Talent Development and non- 
Talent Development schools did not take the eleventh-grade state standards assessment test 
within four years of high school. Impact estimates cannot be reliably calculated for the small 
percentage of students that did take the assessments. The evaluation cannot assess the impact of 
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The Talent Development Evaluation 
Figure 8 

Impacts on the Percentage of Students Earning Basic Academic Curriculum 
for Repeating Ninth-Grade Students 
at the End of the Second Year of High School 
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Year 1 Cohort 



Year 2 Cohort 



Year 3 Cohort 



Baseline averages: Talent Development =15.4; Non-Talent Development = 17.3 
SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 



NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent 
Development high schools. Students in the sample were included on the district's transcript and attendance 
records. The sample excludes students who did not attempt at least one credit during a given school year. 
Repeating 9th-grade students were defined as students whose records indicate that they were in the 9th grade in 
the year under study and were also in the 9th grade in the previous year's administrative data file. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars 
represent the deviations from baseline of the non-Talent Development comparison schools. The deviations were 
calculated as the change in outcome level from the three-year pre-implementation baseline average. 

The impact at follow-up was calculated as the difference in deviations from the baseline average between 
Talent Development schools and non-Talent Development schools. Impacts are pooled across five clusters of 
Talent Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development 
implementation. (The first calendar year of implementation varied by school cluster.) Year 2 Cohort includes 
students who began 9th grade during the second year of Talent Development implementation. Year 3 Cohort 
includes students who began 9th grade during the third year of Talent Development implementation. 

A two-tailed t-test was applied to the impacts. Standard errors and statistical significance levels are adjusted to 
account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

"Basic academic curriculum" is a designation indicating that a student completed a basic academic curriculum, 
earning at least five credits during the 9th grade, with three of those credits being in mathematics, English, and 
science. Because this designation relied on the district’s course-credit code, which does not distinguish between 
elective courses and required courses, the designation may include credits that students earned for elective 
courses or for required courses. Some elective courses did not meet the district's new 1998-1999 promotion 
requirements, but all of them counted toward total credits earned. 
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The Talent Development Evaluation 
Figure 9 

Impacts on the Percentage of Students Promoted to Tenth Grade 
for Repeating Ninth-Grade Students 
at the End of the Second Year of High School 




Baseline averages: Talent Development schools = 46.1; Non-Talent Development schools = 47.9 

SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent 
Development high schools. Students in the sample were included on the district's transcript and attendance 
records. The sample excludes students who did not attempt at least one credit during a given school year. 
Repeating 9th-grade students were defined as students whose records indicate that they were in the 9th grade in the 
year under study and were also in the 9th grade in the previous year's administrative data file. 

The black bars represent the deviations from baseline of the Talent Development schools. The white bars 
represent the deviations from baseline of the non-Talent Development comparison schools. The deviations were 
calculated as the change in outcome level from the three-year pre-implementation baseline average. 

The impact at follow-up was calculated as the difference in deviations from the baseline average between Talent 
Development schools and non-Talent Development schools. Impacts are pooled across five clusters of Talent 
Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development 
implementation. (The first calendar year of implementation varied by school cluster.) Year 2 Cohort includes 
students who began 9th grade during the second year of Talent Development implementation. 

A two-tailed t-test was applied to the impacts. Standard errors and statistical significance levels are adjusted to 
account for cohort effects. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

For the purposes of this analysis, 9th-grade students were considered to have been promoted to 10th grade if 
they were listed as lOth-graders in the next year's administrative data file. 
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Talent Development on the eleventh-grade state standards assessment for repeating ninth- 
grade students. 

A Look at Graduation: Impacts for Students in the Two Earliest- 
Implementing Talent Development High Schools 

This section will discuss findings for students in the two earliest-implementing Talent 
Development high schools. Five years of follow-up data are available for these schools, which 
allows four- and five-year high school transcripts to be assembled for ninth-grade students in 
the earliest cohorts in these schools. The impact of Talent Development on credits required for 
graduation and graduation rates is assessed. Eleventh-grade test-score data are also available for 
three cohorts of students in the earliest-implementing schools. The findings provide some in- 
sight as to Talent Development’s impact on graduation rates and offer another look at its impact 
on test scores, but they may not be representative of likely outcomes for other Talent Develop- 
ment schools in the district. The implementation history for these early-implementing schools 
shares much in common with the other Talent Development schools in the district, but there are 
some unique aspects to the reform’s rollout in these schools. There also appears to be a some- 
what different pattern of impacts for outcomes in the first three years of high school for students 
in this subset of schools as compared to the full sample of five schools. 

• The two earliest-implementing schools are similar to later-implementing 
Talent Development schools, but implementation of the reform in these 
schools benefited from intense support from the model developers and 
consistent leadership at the school level. 

Though these two schools are similar to later-implementing schools — low-performing 
high schools in low-income neighborhoods serving mostly poor students — their implementa- 
tion histories differ somewhat from those schools that adopted Talent Development later. As the 
first two schools, in an implementation process that was evolutionary, they benefited from the 
undivided and intensive efforts of Talent Development implementers, particularly from some of 
the most experienced staff that had worked directly with the first implementing school in Balti- 
more. Though Talent Development implementation staff managed the planning and implemen- 
tation process in all the Philadelphia schools, their presence in these two schools was probably 
more intensive and extended. 

As noted earlier, the principals of these two schools, having long recognized the need 
for substantial changes, were quite open to the Talent Development concept of whole-school 
reform and made considerable efforts to install not just the ninth-grade elements, but also to in- 
stitute changes in their existing upper-grade academies and small learning communities as well. 
Both schools also made extensive use of the professional development and training opportuni- 



70 




ties. Especially in the early going (before attrition and turnover somewhat weakened the effect), 
a comparatively high proportion of the faculty and staff working in these schools had been ex- 
posed to Talent Development institutes and conferences. 

Finally, one of these two schools has had the same principal in place from the beginning 
of planning for Talent Development until the present. The other experienced changes in leadership 
but offset these changes somewhat by having the same Talent Development staff leader in place 
since the program was first implemented. In both cases, an important element of continuity proba- 
bly contributed in some way to any differences in level of impacts that these schools produced. 

• The estimated impacts of Talent Development on attendance, course 
credit outcomes, and promotion outcomes in the first three years of high 
school for the two earliest-implementing high schools were greater than 
the estimated impacts for the sample of all five high schools. 

Table 8 shows the baseline levels for attendance, credits earned, enrollment status, and 
test scores for first-time ninth-grade students in the two earliest-implementing Talent Develop- 
ment schools. 51 Comparing Table 8 with Table 5 illustrates that, in the baseline period, the out- 
come levels for cohorts of first-time ninth-grade students in the two earliest-implementing 
schools were similar to outcome levels for cohorts of first-time ninth-grade students in the full 
sample of five schools. However, in the follow-up period, the estimated impact of Talent De- 
velopment on credits earned, promotion rates, and test scores in the first three years of high 
school appears to be greater in the earliest-implementing schools than for the full sample of five 
schools. 



The first two pages of Table 9 provide a summary of the impact estimates for the first 
three years of high school for students in the two earliest-implementing schools. For this subset 
of schools as for the full sample, Talent Development produced consistently positive impacts on 
attendance, course credit, and promotion outcomes across several cohorts of first-time ninth- 
grade students. The estimated impact of Talent Development on attendance outcomes was simi- 
lar across cohorts of students both in the earliest-implementing schools and in all five schools. 
However, impact estimates for credits earned in the first, second, and third year of high school 
are consistently larger for each of cohort of students in the two-school sample as compared to 
the five-school sample. The estimated impact of Talent Development on promotion from 



51 Findings for the two early-implementing schools focus on cohorts of first-time ninth-grade students be- 
cause a very limited sample of repeating ninth-grade sUidents was still in school after five years. Nearly 60 
percent of repeating ninth-grade students had exited the school system without graduating by the end of the 
fourth year, and 80 percent had exited the school system without graduating by the end of the fifth year of high 
school. 
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The Talent Development Evaluation 
Table 8 



Baseline Outcome Levels for Ninth-Grade Students in Earliest-Implementing 
Talent Development and Non-Talent Development Comparison Schools 





First-Time 


Repeating 




Ninth-Grade Students 


Ninth-Grade Students 




Talent 


Non-Talent 


Talent 


Non-Talent 




Development 


Development 


Development 


Development 


Outcome 


Schools 


Schools 


Schools 


Schools 


At the end of the first year of high school 

Attendance 11 (%) 










Attendance rate 

Students with an attendance rate of: 


71.1 


73.5 


49.8 


52.3 


90% or higher 


19.7 


23.4 


2.3 


3.7 


80% or lower 


58.7 


54.9 


92.2 


89.1 


Course credits earned 










Total credits earned 


4.10 


4.29 


3.11 


3.58 


5 or more credits for the year 3 (%) 


51.0 


53.0 


27.5 


32.0 


Basic academic curriculunT (%) 


39.8 


42.2 


15.5 


18.4 


At least 1 English credit for the year (%) 


64.0 


63.7 


38.5 


43.4 


At least 1 math credit for the year (%) 


51.2 


62.0 


29.6 


37.8 


At least 1 algebra credit for the year (%) 

At the end of the second year of high school 

Enrollment status' 1 (%) 


24.1 


31.9 


12.1 


14.5 


Enrolled in the 10th grade 


60.8 


62.8 


38.3 


43.5 


Enrolled in the 9th grade 


37.4 


34.9 


55.8 


44.2 


Exited the school system 


1.8 


2.3 


5.9 


12.2 


Course credits earned 










Total credits earned over first two years 
At least 2 credits in English 


7.96 


8.04 


4.57 


5.17 


and at least 2 credits in math (%) 


32.4 


37.5 


9.7 


12.3 


At the end of the third year of high school 










Enrollment status" (%) 










Enrolled in the 11th grade 


45.9 


47.3 


19.6 


21.2 


Enrolled in the 9th or 10th grade 


41.9 


41.7 


41.9 


34.9 


Exited the school system 


12.1 


11.0 


38.4 


43.8 


Course credits earned 










Total credits earned over first three years 
At least 3 credits in English 


11.6 


11.5 


5.8 


6.4 


and at least 3 credits in math (%) 


28.5 


29.9 


7.8 


9.5 


At least 17.5 total credits' (%) 


32.7 


29.7 


10.6 


10.7 



(continued) 
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Table 8 (continued) 



First-Time Repeating 

Ninth-Grade Students Ninth-Grade Students 



Talent 


Non-Talent 


Talent 


Non-Talent 


Development 


Development 


Development 


Development 



Outcome 


Schools 


Schools 


Schools 


Schools 


1 lth-grade PSSA test score 8 


Took the test on time (%) 
For test-takers: 

Math 


34.9 


35.4 


NA 


NA 


Average scaled score 


1,061.5 


1,058.5 


NA 


NA 


At below-basic level (%) 


84.9 


87.2 


NA 


NA 


At basic level (%) 


11.1 


10.0 


NA 


NA 


At or above proficient level (%) 
Reading 


4.0 


2.9 


NA 


NA 


Average scaled score 


1,017.1 


1,025.0 


NA 


NA 


At below-basic level (%) 


78.5 


78.9 


NA 


NA 


At basic level (%) 


16.1 


16.3 


NA 


NA 


At or above proficient level (%) 

At the end of the fourth year of high school 


5.3 


4.6 


NA 


NA 


Enrollment status 11 (%) 


Graduated 


33.6 


34.9 


5.4 


6.8 


Enrolled in the 12th grade but did not graduate 


8.8 


8.6 


4.4 


5.1 


Enrolled in the 9th, 10th, or 1 1th grade 


33.8 


32.6 


33.5 


29.0 


Exited the school system 


23.8 


23.9 


56.7 


58.9 


Course credits earned 


Total credits earned over first three years 
At least 4 credits in English 


14.9 


14.6 


6.3 


7.0 


and at least 4 credits in math (%) 


27.2 


26.1 


3.3 


3.6 


At least 17.5 total credits 1 (%) 


45.9 


45.5 


14.5 


15.4 


Earned core graduation credits 1 (%) 


29.9 


26.6 


2.9 


3.0 



(continued) 



SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia 

NOTES: Sample includes 9th-grade students from two Talent Development high schools and six non-Talent 
Development high schools. Students in the sample were included on the district's transcript and attendance 
records. The sample excludes students who did not attempt at least one credit during a given school year. First- 
time 9th-grade students were defined as students whose records indicate that they were in the 9th grade in the year 
under study and in the 8th grade in the previous year's administrative data file. Repeating 9th-grade students were 
defined as students whose records indicate that they were in the 9th grade in the year under study and were also in 
the 9th grade in the previous year's administrative data file. 

The levels presented are averages taken over the three years preceeding the implementation of Talent 
Development in each Talent Development school and the same three years for the non-Talent Development 
comparison schools. 

“Attendance rates were calculated for each student by dividing the number of days the student was present by 
the total number of days the student was enrolled in a given school year. 
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Table 8 (continued) 

b Until the 1998-1999 school year, 9th-grade students in the district were required to earn four course credits in 
order to be promoted. Beginning in the 1998-1999 school year, minimum requirements for promotion included 
earning at least five credits during the 9th grade, with three of those credits awarded for completing one required 
course in mathematics, one in English, and one in science. In recent years, the distributive requirement has been 
dropped. 

c “Basic academic curriculum” is a designation indicating that a student completed a basic academic curriculum, 
earning at least five credits during the 9th grade, with three of those credits being in mathematics, English, and 
science. Because this designation relied on the district’s course-credit code, which does not distinguish between 
elective courses and required courses, the designation may include credits that students earned for elective courses 
or for required courses. Some elective courses did not meet the district's new 1998-1999 promotion requirements, 
but all of them counted toward total credits earned. 

d For the purposes of this analysis, 9th-grade students were considered to have been promoted to 10th grade if 
they were listed as lOth-graders in the next year's administrative data file. 

Tor the purposes of this analysis, 9th-grade students were considered to have been promoted to 1 1th if they were 
listed as 1 lth-graders in the administrative data file two years after the current year. 

'Students are required to earn 17.5 credits in order to be promoted to the 12th grade. 

Pennsylvania System of Schools Assessment (PSSA) is a criterion-referenced test administered in grades 5, 8, 
and 11, which provides information on student performance on skills and content knowledge specified by the state. 

h For the purposes of this analysis, 9th-grade students were considered to have graduated if so indicated on 
administrative records. Ninth-grade students were considered to have been promoted to 12th grade and not 
graduated if they were listed as 12th-graders in the administrative data file three years after the current year but did 
not have graduate status for that year. 

‘"Earned Core Graduation Credits” is a designation indicating that a student completed an academic curriculum 
of at least 23.5 credits, earning at least four credits in English, at least three credits in math, at least three credits in 
science, and at least three credits in social studies. Because this designation relied on the district's course-credit 
code, which does not distinguish between elective courses and required courses, the designation may include 
credits that students earned for elective courses or for required courses. 
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The Talent Development Evaluation 

Table 9 



Impacts on Outcomes for First-Time Ninth-Grade Students in Earliest-Implementing Schools 









Impact at Follow-Up 








Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Cross-Cohort 


Outcome 


Cohort 


Cohort 


Cohort 


Cohort 


Cohort 


Average 


At the end of the first year of high school 

Attendance 1 * (%) 














Attendance rate 


4.31 


3.68 


4.37 


7.00 ** 


7.15 ** 


5 3 *** 


Impact effect size 
Students with an attendance rate of: 


0.14 


0.12 


0.15 


0.23 ** 


0.24 ** 


0 i 8 *** 


90% or higher 


5.0 


7.5 


8.1 


13.5 *** 


15 y *** 


10.0 *** 


80% or lower 


- 10.1 * 


-6.5 


- 6.8 


-16 7 *** 


-19.4 *** 


-11.9 *** 


Course credits earned 














Total credits earned 


0.83 


0.88 


0.91 


1.44 ** 


0.85 


1 o *** 


Impact effect size 


0.31 


0.33 


0.34 


0.54 ** 


0.32 


q 32 *** 


5 or more credits for the year (%) 


10.2 


7.0 


8.9 


18.2 ** 


9.3 


IQ 2 *** 


Basic academic curriculum 1 * (%) 


13.6 ** 


9.3 


13.5 * 


23 3 *** 


10.0 


13 9 *** 


At least 1 English credit for the year (%) 


8.5 


6.2 


10.7 


10.4 


2.2 


7.6 ** 


At least 1 math credit for the year (%) 


18.5 ** 


17.3 ** 


22 7 *** 


25.5 *** 


15.7 * 


19 9 *** 


At least 1 algebra credit for the year (%) 

At the end of the second year of high school 

Enrollment status * 1 (%) 


33.6 *** 


32.8 ** 


22 2 *** 


41 2 *** 


10.9 


31 1 *** 


Enrolled in the 10th grade 


14.8 * 


9.2 


11.4 


15.8 * 




12.8 *** 


Enrolled in the 9th grade 


-16.6 ** 


-9.7 


-13.9 * 


-16.0 ** 




_14 i *** 


Exited the school system 


1.8 


0.5 


2.4 


0.1 




1.2 
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Table 9 (continued) 









Impact at Follow-Up 








Year 1 


Year 2 


Year 3 


Year 4 


Year 5 


Cross-Cohort 


Outcome 


Cohort 


Cohort 


Cohort 


Cohort 


Cohort 


Average 


Course credits earned 














Total credits earned over first two years 


1.36 * 


0.84 


1.38 * 


l 97 *** 




j 4 *** 


Impact effect size 


0.26 * 


0.16 


0.26 * 


0.38 *** 




0 27 *** 


At least 2 credits in English and at least 2 credits in math (%) 


22.8 *** 


9.1 


21.2 *** 


2 Q 7 *** 




13 4 *** 


At the end of the third year of high school 














Enrollment status' (%) 














Enrolled in the 1 1th grade 


9.0 


- 0.8 


9.6 






5.9 


Enrolled in the 9 th or 10th grade 


-8.5 


-1.3 


-12.3 ** 






_7 3 ** 


Exited the school system 


-0.5 


2.1 


2.7 






1.4 


Course credits earned 














Total credits earned over first three years 


1.31 * 


1.04 


1.91 ** 






j 4 *** 


Impact effect size 


0.17 * 


0.13 


0.25 ** 






0.18 *** 


At least 3 credits in English and at least 3 credits in math (%) 


13.5 *** 


5.3 


16.6 *** 






1 1 g *** 


At least 17.5 total credits* (%) 


3.5 


2.4 


6.2 






4.0 


1 lth-grade PSSA test score 8 














Took the test on time (%) 
For test-takers: 


5.9 


0.3 


6.7 






4.3 


Math 














Average scaled score 


28.7 


29.9 


79 O *** 






45 9 *** 


Impact effect size 


0.24 


0.25 


0.65 *** 






0.38 *** 


At below-basic level (%) 


-7.5 


* 

00 

oo 


-15.6 *** 






- 10.6 *** 


At basic level (%) 


3.8 


7.1 * 


3.4 






4.8 ** 


At or above proficient level (%) 
Reading 


3.6 


1.4 


12 3 *** 






5.8 *** 


Average scaled score 


-24.5 


16.4 


48.7 ** 






13.5 


Impact effect size 


-0.16 


0.11 


0.32 ** 






0.09 


At below-basic level (%) 


5.0 


-4.2 


-5.2 






-1.4 


At basic level (%) 


-3.6 


6.6 


-2.7 






0.1 


At or above proficient level (%) 


-1.5 


-2.3 


7 g *** 






1.3 
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Table 9 (continued) 






Impact at Follow-Up 



Outcome 


Year 1 
Cohort 


Year 2 
Cohort 


Year 3 
Cohort 


Year 4 
Cohort 


Year 5 
Cohort 


Cross-Cohort 

Average 


At the end of the fourth year of high school 














Enrollment status 11 (%) 














Graduated 


8.2 * 


6.9 








7.5 


Enrolled in the 12th grade but did not graduate 


-4.1 


-2.9 








-3.5 


Enrolled in the 9th, 10th, or 1 1th grade 


-4.9 


-8.4 








-6.7 


Exited the school system 


0.8 


4.5 








2.7 


Course credits earned 














Total credits earned over first four years 


1.48 ** 


1.44 * 








l g *** 


Impact effect size 


0.28 ** 


0.28 * 








0.28 *** 


At least 4 credits in English and at least 4 credits in math (%) 


9.2 


9.8 








9.5 ** 


At least 17.5 total credits 1 (%) 


3.1 


4.0 








3.5 


Earned core graduation credits 1 (%) 


3.7 


3.9 








3.8 



(continued) 



SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 



NOTES: Sample includes 9th-grade students from two early-implementing Talent Development high schools and six non-Talent Development high schools. 
Students in the sample were included on the district's transcript and attendance records. The sample excludes students who did not attempt at least one credit 
during a given school year. First-time 9th-grade students were defined as students whose records indicate that they were in the 9th grade in the year under 
study and in the 8th grade in the previous year's administrative data file. 

The impacts at follow-up were calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent 
Development schools. Impacts are pooled across five clusters of Talent Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development implementation in their school. (The first calendar year 
of implementation varied by school cluster.) Year 2 Cohort includes students who began 9th grade during the second year of Talent Development 
implementation in their school. Year 3 Cohort includes students who began 9th grade during the third year of Talent Development implementation in their 
school. Year 4 Cohort includes students who began 9th grade during the fourth year of Talent Development implementation in their school. Year 5 Cohort 
includes students who began ninth grade during the fifth year of Talent Development implementation in their school. 

The impact effect size was calculated for continuous variables by dividing the impact at follow-up by the standard deviation of the outcome for all 9th- 
grade students in the district's nonselective, comprehensive high schools from school years 1996-1997 through 1998-1999. 




Table 9 (continued) 

Estimates are regression-adjusted using ordinary least squares, controlling for 7th-grade math and reading SAT-9 test scores, race, and whether the student 
had repeated a prior grade. 

A two-tailed t-test was applied to the impacts at follow-up. Standard errors and statistical significance levels are adjusted to account for cohort effects. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

‘‘Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was 
enrolled in a given school year. 

b Until the 1998-1999 school year, 9th-grade students in the district were required to earn four course credits in order to be promoted. Beginning in the 1998- 
1999 school year, minimum requirements for promotion included earning at least five credits during the 9th grade, with three of those credits awarded for 
completing one required course in mathematics, one in English, and one in science. In recent years, the distributive requirement has been dropped. 

““Basic academic curriculum” is a designation indicating that a student completed a basic academic curriculum, earning at least five credits during the 9th 
grade, with three of those credits being in mathematics, English, and science. Because this designation relied on the district's course-credit code, which does 
not distinguish between elective courses and required courses, the designation may include credits that students earned for elective courses or for required 
courses. Some elective courses did not meet the district’s new 1998-1999 promotion requirements, but all of them counted toward total credits earned. 

d For the purposes of this analysis. 9th-grade students were considered to have been promoted to 10th grade if they were listed as lOth-graders in the next 
year's administrative data file. 

“For the purposes of this analysis, 9th-grade students were considered to have been promoted to 1 1th grade if they were listed as 1 lth-graders in the 

— 3 

00 administrative data file two years after the current year. 

f 

Students are required to earn 17.5 credits in order to be promoted to the 12th grade. 

^Pennsylvania System of School Assessment (PSSA) is a criterion-referenced test administered in grades 5, 8, and 11, which provides information on 
student performance on skills and content knowledge specified by the state. 

h For the purposes of this analysis, 9th-grade students were considered to have graduated if so indicated on administrative records. Ninth-grade students were 
considered to have been promoted to 12th grade and not graduated if they were listed as 12th-graders in the administrative data file three years after the 
current year but did not have graduate status for that year. 

'“Earned Core Graduation Credits” is a designation indicating that a student completed an academic curriculum of at least 23.5 credits, earning at least four 
credits in English, at least three credits in math, at least three credits in science, and at least three credits in social studies. Because this designation relied on 
the district’s course-credit code, which does not distinguish between elective courses and required courses, the designation may include credits that students 
earned for elective courses or for required courses. 




ninth to tenth grade is also larger for cohorts of students in the two early-implementing schools, 
though promotion rates to eleventh grade are of similar magnitude to those for the full sample of 
five schools. 

• Talent Development improved eleventh-grade math and reading test 
scores for cohorts of students in the two earliest-implementing schools. 

Test-score data were only available for one cohort of students in the full sample of 
schools. For this cohort, Talent Development reduced the percentage of students scoring in the 
bottom performance category on the state standards assessment in math but did not have an im- 
pact on scores on the reading assessment. For the two earliest-implementing schools, data are 
available for three cohorts of first-time ninth-grade students. As with the full sample, a little 
more than one -third of first-time ninth-grade students in the baseline period took the eleventh- 
grade PSSA by their third year of high school. In the earliest-implementing schools, this per- 
centage improved in both Talent Development and non-Talent Development schools in the fol- 
low-up period, but gains in test-taking rates were greater in Talent Development schools. For 
example, the percentage of test-takers rose from 35 percent to 49 percent for students in Talent 
Development schools in the Year 3 Cohort, while the percentage rose from 35 percent to 43 
percent for students in comparison schools. 52 

As with the full sample of schools, most impact estimates for the first cohort of students 
in the earliest-implementing schools are small and not statistically significant. However, across the 
three cohorts, Talent Development improved the average scaled score on the eleventh-grade state 
assessment in math by 46 points. This represents an effect size of 0.38, or 38 percent of the stu- 
dent-level standard deviation for this outcome. At the same time, Talent Development decreased 
the percentage of students scoring in the bottom performance category by 1 1 percentage points 
and increased the percentage of students scoring at or above proficient by 6 percentage points. The 
pattern of impacts in reading was not as consistent. For students in the Year 3 Cohort, Talent De- 
velopment seems to have improved outcomes on the state standards assessment in reading, but 
there is not significant change for students in the first two cohorts. These findings seem to show 
that with consistent and sustained implementation of the model, Talent Development can have a 
positive impact on student achievement in the upper grades of high school. 

• Talent Development improved graduation rates for cohorts of students 
in the two earliest-implementing schools. 



52 Test-taking rates did not improve greatly by the end of the fourth year of high school. At the end of the 
fourth year of high school, only 37 percent of first-time ninth-grade students in the baseline period had taken 
the eleventh-grade PSSA test. 
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In the baseline period, only one-third of first-time ninth-grade students in the earliest- 
implementing schools graduated on time. By the fourth year of high school, 24 percent of stu- 
dents had withdrawn from the school system. The remaining students had been retained in 
grade or had reached the twelfth grade but had not earned the credits necessary for graduation. 
Their comparison schools had a similar baseline pattern. 

In the follow-up period, graduation rates improved by about 4 percentage points for 
students in the Year 1 Cohort in the earliest-implementing Talent Development schools, while 
graduation rates declined by about 4 percentage points for students in comparison schools. The 
difference in these deviations from the baseline period average (8 percentage points), which is 
the estimated impact of Talent Development on graduation rates for the first cohort of students, 
is statistically significant. A similar pattern of findings is found for the Year 2 Cohort, but the 
estimated impact of nearly 7 percentage points is not statistically significant. Talent Develop- 
ment also has a positive impact on graduation rates by the end of the fifth year of high school, 
but graduation rates for first-time ninth-grade students in Talent Development and comparison 
schools do not exceed 42 percent even with an extra year of high school. By the fifth year, an 
equal percentage of students have withdrawn from the school system without graduating. 



Lessons and Implications 

The impacts reported here are promising because they show that Talent Development 
has positive and significant impacts on a range of important outcomes for first-time ninth- 
graders in some of the lowest-performing schools in Philadelphia. Notably, these effects per- 
sisted into the upper grades as students progressed toward graduation. 

Talent Development improved the attendance rates for first-time ninth-grade students 
by 5 percentage points. A greater percentage of first-time ninth-graders regularly attended 
school, and a lower percentage were chronic absentees. Increased attendance seems to have led 
to higher course completion rates. Not only did students in Talent Development schools earn 
more total credits in their first year of high school, but also a greater percentage of Talent De- 
velopment students earned credits in algebra, which is a critical gate-keeping course that is often 
required for both graduation and admission to college. By helping students earn more credits, 
Talent Development had a positive impact on promotion rates from ninth to tenth grade. Gains 
in Talent Development schools exceed those in non-Talent Development comparison schools 
by 8 percentage points. 

Course-credit gains made by students in the first year of high school were sustained 
through students’ second and third years of high school. Again, gains in Talent Development 
schools outpaced gains in comparison schools on indicators of progress toward graduation — 
earning two English and two math credits by the end of the second year of high school and 



80 




earning three English and three math credits by the end of the third year of high school. Still, 
these schools had much room for improvement. Only about half of first-time ninth-grade stu- 
dents were promoted to eleventh grade on time in both Talent Development and non-Talent 
Development schools, and more than three-quarters of students fell into the bottom category on 
the Pennsylvania System of School Assessment (PSSA) in math or reading. 

Talent Development schools also did a better job of improving promotion rates to elev- 
enth grade than their matched comparison schools, and, for a subset of schools, they did a better 
job of improving four-year graduation rates as well. However, only about one- t hird of students 
who began ninth grade in these low-performing high schools graduated on time. Further, it is 
not clear that, even with Talent Development, students graduate with the skills they need for 
work and college. Talent Development did reduce the percentage of students in the bottom 
category of the PSSA in math, but fewer than 10 percent of students who took the tests were 
considered proficient at math or reading. 

The pattern of results in this report stands out from other research on high school re- 
forms because the impacts are consistently positive across several outcomes; they emerged in 
the first year of implementation; they are sustained for successive cohorts of students; and they 
were found across five high schools. These findings are unique in that the impact of the pro- 
gram is assessed as cohorts of students move through high schools, and the study follows ninth- 
grade students through four years of high school. Impacts are estimated using a particularly 
strong methodology that compares the performance of cohorts of students in Talent Develop- 
ment schools not only to the performance of past cohorts of students in the same school, but to 
gains made by cohorts of students in similar schools. All these factors lead to confidence in the 
findings, but there are still questions: 

• How can the magnitude of Talent Development’s impact on student out- 
comes be assessed? 

• What might explain the impacts for first-time ninth-grade students? 

• Why are impacts not as pronounced for upper-grade students? 

• Why do higher rates of course completion fail to lead to greater achievement 
gains? 

• Why is there no impact on students who repeat ninth grade for a full year? 

• How generalizable are these findings to other schools and other districts? 
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The Magnitude of Talent Development’s Impact 

The magnitude of the impacts outlined above can be evaluated in several ways. Effect 
sizes show each impact as a proportion of the student-level standard deviation for each out- 
come. For example, the impact of Talent Development on credits earned in the first year of high 
school is two-thirds of a credit. This corresponds to an effect size of 0.25, or one-quarter of the 
student-level standard deviation for this outcome in the pre-Talent Development period. The 
effect sizes reported for first-time ninth-grade students range between 0.04 and 0.25, with 
somewhat higher effect sizes, ranging between 0.09 and 0.38, reported in findings for the two 
earliest-implementing schools. 

Although no absolute standard exists to define whether a specific effect size is large or 
small, there are some traditional guidelines. They suggest that the effect sizes in this report fall 
in the small-to-moderate categories. More recent analyses suggest that this categorization may 
be too conservative for educational outcomes. It is possible for small-to-moderate effect sizes to 
have substantial significance in education settings. For example, for a typical class of about 500 
first-time ninth-grade students, Talent Development's 25 percentage-point impact on students 
earning a credit in algebra means that the intervention adds 125 students from each high school 
each year to the rolls of those who reach this critical milestone. Also, the 8 percentage-point 
impact on graduation rates translates into 40 more of the 500 ninth-graders in each Talent De- 
velopment school earning a high school diploma each year. 

Further, the findings for student achievement appear to be comparable with or to exceed 
results found in other third-party evaluations of Comprehensive School Reform (CSR) that have 
strong evidence of effectiveness. A meta-analysis of the 29 most widely discussed and dissemi- 
nated comprehensive school-reform models found the average student achievement effect size 
to be 0.15. This drops to 0.12 for studies using a comparison group design, and to 0.09 when a 
third party conducts the evaluation. 53 In this report, the eleventh-grade math achievement effect 
size was 0.12 for the full sample of schools and 0.38 for the two earliest-implementing schools. 

Even successful interventions like Talent Development still need much more power if 
they are to achieve the goal of preparing all students for graduation, postsecondary education, 
and employment. Even with the gains described in this report, in a high school of 500 first-time 
ninth-graders, the average ninth-grader will still miss about 40 days of school, 150 students will 
not be promoted to the tenth grade on time, and more than half will not be ready to graduate 
within four years. 



53 

Borman, Hewes, Overman, and Brown, 2003. 
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Impacts for First-Time Ninth-Grade Students in the First Year 
of High School 

First-time ninth-grade students seem to derive the greatest benefit from the Talent De- 
velopment model. They have better attendance rates and more credits earned, partly because of 
the special focus of the Ninth Grade Success Academy. While Talent Development’s impact on 
attendance for first-time ninth-graders is straightforward, interpreting course credit findings is 
more complex. 

The extended block schedule used in Talent Development schools provides ninth-grade 
students the opportunity to earn eight credits in their first year of high school. A more traditional 
schedule, common to most other high schools in the district, allows students to earn six or seven 
credits in their first year of high school. This difference in opportunity is clearly part of the posi- 
tive effect Talent Development had on credits earned in the first year of high school. The ques- 
tion, then, seems to be whether the extra two-thirds of a credit earned in Talent Development 
schools really was worth something or whether it translated to just more “seat time” (that is, 
unproductive classroom time) for students? 

The answer may lie in the greater percentage of students earning academic credits, par- 
ticularly in algebra. The extended block schedule accommodates double -dosing of English and 
math for ninth-grade students in Talent Development schools. The Talent Development catch- 
up courses scheduled in the first semester are designed to prepare students for English 1 and 
algebra in the second semester. The catch-up courses, though electives, count toward total cred- 
its. The impacts on credits earned in English and math would be less impressive if they simply 
reflected students passing these transitional courses, rather than courses that meet distributive 
requirements for graduation. However, the large impacts on the percentage of students earning a 
credit in algebra, as well as improved attendance rates, imply a more promising story. Not only 
did Talent Development provide more opportunities to earn credits, the intervention also pro- 
vided higher-quality learning opportunities that led to students meeting promotion and gradua- 
tion requirements. 

Limitations in the Upper Grades 

The relatively modest impacts for upper-grade students in the Philadelphia schools may 
arise from two factors. First, as discussed in the implementation findings of the report, the upper 
grades, with their history of pre-existing academies and small learning communities, proved more 
resistant to the implementation efforts of Talent Development; fewer changes at that level were 
achieved. It may be that the intended effects of Talent Development were not produced because 
implementation in the upper grades was not as robust as at the ninth-grade level, where the Suc- 
cess Academies were successfully put into place. The implementation changes at the lower grades 
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produced a greater difference in services between the Talent Development and comparison 
schools. Thus, the measured impacts at the upper grades would have appeared smaller. 

The second factor may be that the strong effects that Talent Development produced on 
attendance, course-taking, and promotion in lower grades worked to hold more students in 
school for a longer period of time, students who otherwise might have underperformed, stopped 
coming to school, or not been promoted on time. Thus in Talent Development schools, the mix 
of upper-grade students would be more heavily weighted with students who, in non-Talent De- 
velopment schools, would likely have left or not been promoted. The net result would presuma- 
bly be somewhat more challenging upper-grade students whose overall performance might be 
expected to be lower than in a non-Talent Development school, where more of the challenging 
students would have already left. By virtue of the effects of Talent Development at lower 
grades, the upper-grade populations in Talent Development schools might have had lower com- 
parative performance simply because they ended up with a higher percentage of lower- 
performing students. 

Achievement Gains 

Given the relatively large gains in course credits, the student achievement findings seem 
surprising. For the full sample of schools, average scaled scores on the eleventh-grade PSSA did 
not significantly improve. Nor was there great change in the percentage of students at the be- 
low-basic, basic, and at-or-above-proficient performance levels, which are based on the scaled 
scores. The largest impact for the full sample was a 6 percentage-point reduction in the percent- 
age of students scoring in the below-basic category in math. There is some precedent for the 
failure of high levels of course-taking to translate to high-proficiency scores. 54 However, the 
explanation is likely a combination of several factors. 

First, it is important to keep in mind that about 85 percent of students in Talent Devel- 
opment and non-Talent Development schools scored in the below-basic category on the PSSA 
in math and about 75 percent scored in the below-basic category in reading. The PSSA became 
an increasingly high-stakes test in the district, and all schools — both Talent Development and 
non-Talent Development — were under considerable pressure to improve test scores. In fact, 
the district provided all schools with materials to help students prepare for the PSSA. Low- 
performing high schools were encouraged to identify students on the cusps of proficiency levels 
— particularly, from basic to proficient — to help boost school-wide performance. Given simi- 
lar test-prep treatments across schools, it may not be so surprising that the difference in gains 
between Talent Development and non-Talent Development schools (which is the estimated im- 
pact of the intervention) is small. 



54 Lee, Croninger, and Smith, 1997. 
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Second, the full sample findings are for only one cohort of students — the cohort that 
consisted of first-time ninth-graders during the first year of Talent Development in their schools. 
Upper-grade components of the model, which are not as extensive as the ninth-grade compo- 
nents even when fully implemented, may not have been well-instituted in Talent Development 
schools as the first cohort moved into tenth and eleventh grade. Findings for the two earliest- 
implementing schools have more promising results on the PSSA, particularly for the third co- 
hort to move through Talent Development schools. These two schools had stronger implemen- 
tation histories and consistent school leadership, so their results may be a combination of better 
Talent Development implementation and school effects. 

Repeating Ninth-Grade Students 

During the baseline period prior to Talent Development implementation, about one- 
third of ninth-grade students in the analysis were repeating the grade for at least the first time. 
Many of these students were chronic absentees, missing an average of more than 80 days of 
school per year. The average repeating ninth-grader had less than a 50 percent chance of being 
promoted to the tenth grade the following year. For this group, Talent Development produced 
more than a 5 percentage-point increase in average attendance rates. Also, the Talent Develop- 
ment high schools saw notable increases in the credits earned by their repeating ninth-graders, 
although these trends were present in the non-Talent Development schools as well. Overall, 
however, Talent Development actually increased the likelihood of a repeating ninth-grader leav- 
ing the school system before the end of his or her fourth year of high school. 

The mixed results for repeating ninth-grade students may be an artifact of two interact- 
ing phenomena. First, as discussed earlier, Talent Development produced an increase in the rate 
at which first-time ninth-graders were being promoted to the tenth grade. This reduced the 
population of repeaters in the Talent Development schools. The remaining group of repeaters 
might have been somewhat more disadvantaged than the typical repeater in the baseline period 
or in the comparison school in a way not easily measured and, on average, was likely to require 
more intensive services to help them get promoted. At the same time, the implementation analy- 
sis indicated that, in Philadelphia, Talent Development did not execute specialized interventions 
to meet the needs of repeating ninth-grade students. 

Generalizability 

Since the findings in this report reflect just five schools in one city, the question of how 
generalizable they are — how much they might reflect implementation of the Talent Develop- 
ment model in other high schools in other cities — cannot be answered definitively. That is par- 
ticularly true because of several distinctive features that characterized the implementation proc- 
ess in Philadelphia. First, the school district at that time took a mostly hands-off posture with 
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respect to Talent Development. They sanctioned the adoption of the model by high schools, but 
provided no official support or recognition. The potential advantages of official recognition — 
greater authority to institute changes in the schools, control staffing and leadership, and com- 
mand funding and resources — were thus not available to the implementers in Philadelphia. 

This limitation was offset by two factors. Implementers had a comparatively free hand in 
how they went about their work. Also, particularly in the early stages, the staff at CRESPAR and 
the local intermediary could concentrate their efforts on the implementation process in an unusu- 
ally intensive way. It is certainly also true that implementation in Philadelphia, which was a sort of 
test case city for expansion of the Talent Development model, had an evolutionary character, and 
that some unevenness in the process across schools resulted as CRESPAR and the local interme- 
diary worked to codify and routinize the process of new school implementation. 

The fair conclusion seems to be that the responses to school district disinterest and the 
challenges of new city expansion of the model, effective as those responses were in Philadel- 
phia, can probably be equaled or bettered in other cities. CRESPAR can now bring the lessons 
and experience of Philadelphia to bear in other communities. If the Philadelphia experience is 
any indication, other communities would, in all likelihood, benefit from the direct support of 
school district leadership. 

It is important to remember that the initial positive results from Talent Development re- 
quired significant extra funds and very demanding changes to school organization, instruction, and 
teacher support. It is not clear that the results from Philadelphia can be duplicated or sustained 
without these important elements. However, with a sustained investment in specific expertise for 
dealing with poorly prepared students and weak learning environments, built on reasonable fidel- 
ity to the tenets and components of Talent Development, it is reasonable to expect that some real 
progress can be made in some of the lowest-performing high schools in the country. 
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Appendix A 

Tables for First-Time Ninth-Grade Students 




The Talent Development Evaluation 

Appendix Table A.l 

Outcome Levels and Impacts on Outcomes 
Measured at the End of the First Year of High School 
For First-Time Ninth-Grade Students, Follow-Up Year by Cohort 

I. Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 



Outcome 


Year 1 
Cohort 


Year 2 
Cohort 


Year 3 
Cohort 


Year 1 
Cohort 


Year 2 
Cohort 


Year 3 
Cohort 




Attendance " 1 (%) 


Attendance rate 


75.6 


78.9 


77.7 


76.6 


76.0 


75.5 




Deviation from baseline 


2.8 ** 


6.1 *** 


4 q *** 


0.1 


-0.5 


- 1.0 




Students with an attendance rate of: 


90% or higher 


26.5 


26.8 


27.5 


21 A 


23.9 


24.1 




Deviation front baseline 


5.0 * 


5.4 * 


6.0 * 


0.1 


-3.3 ** 


-3.2 


* 


80% or lower 


51.4 


47.1 


46.6 


49.0 


52.6 


52.8 




Deviation from baseline 


- 6.0 * 


- 10.2 *** 


-10 7 *** 


-0.5 


3.2 * 


3.4 


* 


Course credits earned 


Total credits earned 


5.22 


5.07 


5.29 


4.84 


4.79 


4.85 




Deviation from baseline 


0 91 *** 


0.76 ** 


0.98 *** 


0.22 


0.18 


0.23 




5 or more credits for the year b (%) 


63.5 


59.2 


63.7 


60.4 


58.6 


59.4 




Deviation from baseline 


10.2 *** 


5.8 * 


10.3 *** 


2.6 


0.9 


1.7 




Basic academic curriculum 1 " (%) 


56.6 


52.7 


56.6 


51.1 


50.3 


52.5 




Deviation from baseline 


13 2 *** 


9 2 *** 


13 i *** 


3.5 * 


2.7 


4.9 


** 


At least 1 English credit for the year (%) 


76.1 


75.0 


79.2 


68.9 


69.3 


72.9 




Deviation from baseline 


10.4 *** 


9 3 *** 


13.5 *** 


1.0 


1.4 


5.0 


** 


At least 1 math credit for the year (%) 


71.6 


71.0 


74.1 


69.1 


69.1 


70.6 




Deviation from baseline 


15.5 *** 


14 9 *** 


18 i *** 


4.0 * 


4.1 * 


5.6 


** 


At least 1 algebra credit for the year (%) 


57.4 


61.7 


64.1 


52.4 


44.3 


49.5 




Deviation from baseline 


24 3 *** 


28.6 *** 


31.0 *** 


7.2 * 


- 1.0 


4.2 





(continued) 




Appendix Table A.l (continued) 







II. Impact of Talent Development 




Outcome 


Year 1 
Cohort 


Year 2 
Cohort 


Year 3 
Cohort 


Attendance ' 1 (%) 


Attendance rate 


2.8 * 


6 7 *** 


5 9 *** 


Students with an attendance rate of: 


90% or higher 


4.9 


8.8 ** 


9 2 *** 


80% or lower 


-5.5 


_13 4 *** 


_14 i *** 


Course credits earned 


Total credits earned 


0.69 * 


0.58 


0.74 ** 


5 or more credits for the year 6 (%) 


7.5 * 


5.0 


8.6 ** 


Basic academic curriculum 0 (%) 


97 ** 


6.5 


8.2 ** 


At least 1 English credit for the year (%) 


9 4 ** 


7 9** 


8.5 ** 


At least 1 math credit for the year (%) 


11.5 ** 


10.8 ** 


22 4 *** 


At least 1 algebra credit for the year (%) 


17 i ** 


29.6 *** 


26.8 *** 



(continued) 



SOURCE: MDRC calculations from individual students" school records from the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent Development high schools. Students in the sample 
were included on the district's transcript and attendance records. The sample excludes students who did not attempt at least one credit during a given school 
year. First-time 9th-grade students were defined as students whose records indicate that they were in the 9th grade in the year under study and in the 8 th grade in 
the previous year's administrative data file. 

The impacts at follow-up were calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent 
Development schools. Impacts are pooled across five clusters of Talent Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development implementation in their school. (The first calendar year of 
implementation varied by school cluster.) Year 2 Cohort includes students who began 9th grade during the second year of Talent Development implementation 
in their school. Year 3 Cohort includes students who began 9th grade during the third year of Talent Development implementation in their school. 

Panel I: Each outcome has two rows of data. The first row shows the average level for that outcome in each cohort. The second row shows the outcome 
level's average deviation from the three-year pre-implementation baseline average in each cohort. 

Panel II: Each outcome has a single row of data that shows the estimated impact of Talent Development for each cohort. The impact at follow-up was 
calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent Development schools. 

Estimates are regression-adjusted using ordinary least squares, controlling for 7th-grade math and reading SAT-9 test scores, race, and whether the student 
had repeated a prior grade. 




Appendix Table A.l (continued) 

A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I, 
and to the impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

a Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was 
enrolled in a given school year. 

b Until the 1998-1999 school year, 9th-grade students in the district were required to earn four course credits in order to be promoted. Beginning in the 1998- 
1999 school year, minimum requirements for promotion included earning at least five credits during the 9th grade, with three of those credits awarded for 
completing one required course in mathematics, one in English, and one in science. In recent years, the distributive requirement has been droped. 

‘“Basic academic curriculum” is a designation indicating that a student completed a basic academic curriculum, earning at least five credits during the 9th 
grade, with three of those credits being in mathematics, English, and science. Because this designation relied on the district’s course credit code, which does not 
distinguish between elective courses and required courses, the designation may include credits that students earned for elective courses or for required courses. 
Some elective courses did not meet the district’s new 1998-1999 promotion requirements, but all of them counted toward total credits earned. 




The Talent Development Evaluation 
Appendix Table A.2 

Outcome Levels and Impacts on Outcomes 
Measured at the End of the Second Year of High School 
for First-Time Ninth-Grade Students, Follow-Up Results by Cohort 



I. Outcome Levels Compared with Baseline Average 
Talent Development Schools Non-Talent Development Schools 

Year 1 Year 2 Year 3 Year 1 Year 2 Year 3 



Outcome Cohort Cohort Cohort Cohort Cohort Cohort 



Enrollment status 1 * (%) 












Enrolled in the 10th grade 


66.3 


69.1 




59.7 


63.6 


Deviation from baseline 


73 ** 


10 i *** 




- 1.2 


2.7 


Enrolled in the 9th grade 


31.3 


29.3 




38.5 


35.0 


Deviation from baseline 


.7 g ** 


_9 g *** 




1.6 


-1.9 


Exited the school system 


2.4 


1.6 




1.8 


1.4 


Deviation from baseline 


0.5 


-0.3 




-0.4 


- 0.8 


Course credits earned 












Total credits earned over first two years 


9.51 


9.47 




8.97 


9.11 


Deviation from baseline 


1 23 *** 


1 19 *** 




0.29 


0.43 ** 


At least 2 credits in English 












and at least 2 credits in math (%) 


55.8 


55.3 




44.2 


49.3 


Deviation from baseline 


19 9 *** 


19 5 *** 


II. Impact of Talent Development 


3.5 ** 


8.5 *** 






Year 1 


Year 2 




Year 3 


Outcome 




Cohort 


Cohort 




Cohort 


Enrollment status'* (%) 












Enrolled in the 10th grade 




8.5 ** 


7.4 * 






Enrolled in the 9th grade 




-9.5 ** 


-7.9 * 






Exited the school system 




0.9 


0.5 






Course credits earned 












Total credits earned over first two years 
At least 2 credits in English 




0.93 ** 


0.76 * 






and at least 2 credits in math (%) 




16.4 *** 


IQ 9 *** 







(continued) 




Appendix Table A.2 (continued) 



SOURCE: MDRC calculations from individual students’ school records front the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent Development high schools. Students in the sample 
were included on the district's transcript and attendance records. The sample excludes students who did not attempt at least one credit during a given school 
year. First-time 9th-grade students were defined as students whose records indicate that they were in the 9th grade in the year under study and in the 8th grade in 
the previous year's administrative data file. 

The impacts at follow-up were calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent 
Development schools. Impacts are pooled across five clusters of Talent Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development implementation in their school. (The first calendar year of 
implementation varied by school cluster.) Year 2 Cohort includes students who began 9th grade during the second year of Talent Development implementation 
in their school. Year 3 Cohort includes students who began 9th grade during the third year of Talent Development implementation in their school. 

Panel I: Each outcome has two rows of data. The first row shows the average level for that outcome in each cohort. The second row shows the outcome 
level's average deviation from the three-year pre-implementation baseline average in each cohort. 

Panel II: Each outcome has a single row of data that shows the estimated impact of Talent Development for each cohort. The impact at follow-up was 
calculated as the difference in deviations front the baseline average between Talent Development schools and non-Talent Development schools. 

Estimates are regression-adjusted using ordinary least squares, controlling for 7th-grade math and reading SAT-9 test scores, race, and whether the student 
£2 had repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I, 
and to the impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

J For the purposes of this analysis, 9th-grade students were considered to have been promoted to 10th grade if they were listed as lOth-graders in the next 
year's administrative data file. 




VO 



The Talent Development Evaluation 

Appendix Table A.3 

Outcome Levels and Impacts on Outcomes 
Measured at the End of the Third Year of High School 
for First-Time Ninth-Grade Students, Follow-Up Results by Cohort 

I. Outcome Levels Compared with Baseline Average 

Talent Development Schools Non-Talent Development Schools 

Year 1 Year 2 Year 3 Year 1 Year 2 Year 3 

Outcome Cohort Cohort Cohort Cohort Cohort Cohort 



Enrollment status ' 1 (%) 



Enrolled in the 1 1th grade 


53.2 




48.6 


Deviation from baseline 


6.0 * 




-0.4 


Enrolled in the 10th grade 


39.2 




44.4 


Deviation from baseline 


-5.0 




2.3 


Exited the school system 


7.6 




7.0 


Deviation from baseline 


- 1.0 




- 1.8 


Course credits earned 








Total credits earned over first three years 


13.2 




12.8 


Deviation from baseline 


1 3 *** 




0.4 * 


At least 3 credits in English 








and at least 3 credits in math (%) 


45.8 




38.2 


Deviation from baseline 


15.1 *** 




4 ^ *** 


At least 17.5 total credits ' 3 (%) 


38.5 




37.7 


Deviation from baseline 


4.7 * 




2.8 * 








II. Impact of Talent Development 


Outcome 


Year 1 Cohort 


Year 2 Cohort Year 3 Cohort 


Enrollment status 11 (%) 








Enrolled in the 1 1th grade 




6.5 * 




Enrolled in the 10th grade 




-7.3 * 




Exited the school system 




0.8 




Course credits earned 








Total credits earned over first three years 




0.9 * 




At least 3 credits in English 








and at least 3 credits in math (%) 




10.5 *** 




At least 17.5 total credits ' 3 (%) 




1.9 





(continued) 






Appendix Table A.3 (continued) 



SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent Development high schools. Students in the sample 
were included on the district's transcript and attendance records. The sample excludes students who did not attempt at least one credit during a given school 
year. First-time 9th-grade students were defined as students whose records indicate that they were in the 9th grade in the year under study and in the 8th grade in 
the previous year's administrative data file. 

The impacts at follow-up were calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent 
Development schools. Impacts are pooled across five clusters of Talent Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development implementation in their school. (The first calendar year of 
implementation varied by school cluster.) Year 2 Cohort includes students who began 9th grade during the second year of Talent Development implementation 
in their school. Year 3 Cohort includes students who began 9th grade during the third year of Talent Development implementation in their school. 

Panel I: Each outcome has two rows of data. The first row shows the average level for that outcome in each cohort. The second row shows the outcome 
level's average deviation from the three-year pre-implementation baseline average in each cohort. 

Panel II: Each outcome has a single row of data that shows the estimated impact of Talent Development for each cohort. The impact at follow-up was 
calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent Development schools. 

Estimates are regression-adjusted using ordinary least squares, controlling for 7th-grade math and reading SAT-9 test scores, race, and whether the student 
so had repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I, 
and to the impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

a For the purposes of this analysis, 9th-grade students were considered to have been promoted to 1 1th grade if they were listed as 1 lth-graders in the 
administrative data file two years after the current year. 

b Students are required to earn 17.5 credits in order to be promoted to the 12th grade. 




The Talent Development Evaluation 

Appendix Table A.4 

Outcome Levels and Impacts on Test Score Outcomes 
Measured at the End of the Third Year of High School 
for First-Time Ninth-Grade Students, Follow-Up Results by Cohort 



Outcome 


I. Outcome Levels Compared with Baseline Average 
Talent Development Schools Non-Talent Development Schools 


Year 1 
Cohort 


Year 2 Year 3 

Cohort Cohort 


Year 1 
Cohort 


Year 2 Year 3 

Cohort Cohort 


1 lth-grade PSSA test score 11 










Took the test on time (%) 


45.3 




44.7 




Deviation from baseline 


g ] *** 




4 7 *** 




For test-takers: 










Math 










Average scaled score 


1.097.5 




1,089.3 




Deviation from baseline 


33.6 *** 




19.6 *** 




At below basic level (%) 


75.8 




79.6 




Deviation from baseline 


-10 1 *** 




_3 Q ** 




At basic level (%) 


16.1 




13.8 




Deviation from baseline 


4.7 ** 




1.4 




At or above proficient level (%) 


8.0 




6.6 




Deviation from baseline 


5 4 *** 




2.4 ** 




Reading 










Average scaled score 


1.048.0 




1,062.9 




Deviation from baseline 


11.5 




]7 7 ** 




At below basic level (%) 


70.5 




68.4 




Deviation from baseline 


-5.2 




-5.5 ** 




At basic level (%) 


21.2 




21.5 




Deviation from baseline 


2.6 




2.4 




At or above proficient level (%) 


8.4 




10.1 




Deviation from baseline 


2.5 




3 2 *** 





(continued) 




Appendix Table A.4 (continued) 






II. Impact of Talent Development 




Year 1 


Year 2 


Year 3 


Outcome 


Cohort 


Cohort 


Cohort 


1 lth-grade PSSA test score 2 








Took the test on time (%) 


3.5 






For test takers: 








Math 








Average scaled score 


14.1 






At below-basic level (%) 


-6.2 * 






At basic level (%) 


3.2 






At or above proficient level (%) 


3.0 






Reading 








Average scaled score 


-6.2 






At below-basic level (%) 


0.4 






At basic level (%) 


0.3 






At or above proficient level (%) 


-0.7 







(continued) 



SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent Development high schools. Students in the sample 
were included on the district's transcript and attendance records. The sample excludes students who did not attempt at least one credit during a given school 
year. First-time 9th-grade students were defined as students whose records indicate that they were in the 9th grade in the year under study and in the 8th grade in 
the previous year's administrative data file. 

The impacts at follow-up were calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent 
Development schools. Impacts are pooled across five clusters of Talent Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development implementation in their school. (The first calendar year of 
implementation varied by school cluster.) Year 2 Cohort includes students who began 9th grade during the second year of Talent Development implementation 
in their school. Year 3 Cohort includes students who began 9th grade during the third year of Talent Development implementation in their school. 

Panel I: Each outcome has two rows of data. The first row shows the average level for that outcome in each cohort. The second row shows the outcome 
level's average deviation from the three-year pre-implementation baseline average in each cohort. 

Panel II: Each outcome has a single row of data that shows the estimated impact of Talent Development for each cohort. The impact at follow-up was 
calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent Development schools. 

Estimates are regression-adjusted using ordinary least squares, controlling for 7th-grade math and reading SAT-9 test scores, race, and whether the student 
had repeated a prior grade. 




Appendix Table A.4 (continued) 



A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I, 
and to the impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

‘‘Pennsylvania System of School Assessment (PSSA) is a criterion-referenced test administered in grades 5. 8, and 11, which provides information on student 
performance on skills and content knowledge specified by the state. 
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The Talent Development Evaluation 
Appendix Table B.l 

Outcome Levels and Impacts on Outcomes 
Measured at the End of the Second Year of High School 
for Repeating Ninth-Grade Students, Follow-Up Results by Cohort 





I. Outcome Levels Compared with Baseline Average 
Talent Development Schools Non-Talent Development Schools 


Outcome 


Year 1 
Cohort 


Year 2 
Cohort 


Year 3 
Cohort 


Year 1 
Cohort 


Year 2 
Cohort 


Year 3 
Cohort 


Attendance 3 (%) 


Attendance rate 


60.8 


67.4 


60.9 


59.9 


60.2 


58.4 


Deviation from baseline 


4.8 * 


\ 1 4 *** 


4.9 * 


1.8 


2.1 


0.3 


Students with an attendance rate of: 


90% or higher 


5.8 


8.5 


5.2 


6.2 


5.0 


4.6 


Deviation from baseline 


1.0 


37 ** 


0.3 


1.0 


- 0.2 


- 0.6 


80% or lower 


82.6 


76.3 


86.1 


81.5 


83.5 


83.7 


Deviation from baseline 


-4.4 


-10 7 *** 


-0.9 


-3.1 


- 1.1 


-0.9 


Course credits earned 


Total credits earned 


4.82 


5.36 


4.32 


5.44 


5.62 


5.10 


Deviation from baseline 


0.37 


0.91 ** 


-0.13 


0.67 *** 


0.85 *** 


0.32 


5 or more credits for the year b (%) 


42.3 


49.0 


38.3 


52.4 


54.7 


49.2 


Deviation from baseline 


0.2 


6.8 


-3.8 


7 g *** 


Q g *** 


4.3 * 


Basic academic curriculum 3 (%) 


26.3 


33.6 


27.5 


33.3 


36.6 


35.0 


Deviation from baseline 


1.1 


8.4 ** 


2.4 


6.3 *** 


9.6 *** 


g Q *** 


At least 1 English credit for the year (%) 


55.9 


62.2 


59.0 


59.9 


63.4 


62.9 


Deviation from baseline 


4.5 


10.8 *** 


7.6 * 


^ 4 *** 


10.0 *** 


9 5 *** 


At least 1 math credit for the year (%) 


44.5 


54.4 


52.8 


58.2 


60.7 


59.8 


Deviation from baseline 


3.4 


13.3 *** 


1 1 7 *** 


10.8 *** 


13 2 *** 


12 3 *** 


At least 1 algebra credit for the year (%) 


26.9 


37.2 


39.2 


39.9 


41.0 


39.0 


Deviation from baseline 


4.9 


15 2 *** 


j7 2 *** 


13.0 *** 


14 i *** 


12 i *** 



(continued) 







Appendix Table B.l (continued) 







II. Impact of Talent Development 




Outcome 


Year 1 
Cohort 


Year 2 
Cohort 


Year 3 
Cohort 


Attendance " 1 (%) 


Attendance rate 


3.0 


9 3 *** 


4.6 


Students with an attendance rate of: 


90% or higher 


0.0 


3.9 * 


0.9 


80% or lower 


-1.3 


-9.5 ** 


0.1 


Course credits earned 


Total credits earned 


-0.3 


0.1 


-0.4 


5 or more credits for the year b (%) 


-7.3 


-3.0 


- 8.1 * 


Basic academic curriculum 0 (%) 


-5.2 


- 1.2 


-5.6 


At least 1 English credit for the year (%) 


-1.9 


0.9 


-1.9 


At least 1 math credit for the year (%) 


-7.4 


0.1 


- 0.6 


At least 1 algebra credit for the year (%) 


- 8.1 


1.2 


5.1 



(continued) 



SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent Development high schools. Students in the sample 
were included on the district's transcript and attendance records. The sample excludes students who did not attempt at least one credit during a given school 
year. Repeating 9th-grade students were defined as students whose records indicate that they were in the 9th grade in the year under study and were also in the 
9th grade in the previous year's administrative data file. 

The impacts at follow-up were calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent 
Development schools. Impacts are pooled across five clusters of Talent Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development implementation in their school. (The first calendar year of 
implementation varied by school cluster.) Year 2 Cohort includes students who began 9th grade during the second year of Talent Development implementation 
in their school. Year 3 Cohort includes students who began 9th grade during the third year of Talent Development implementation in their school. 

Panel I: Each outcome has two rows of data. The first row shows the average level for that outcome in each cohort. The second row shows the outcome 
level's average deviation from the three-year pre-implementation baseline average in each cohort. 

Panel II: Each outcome has a single row of data that shows the estimated impact of Talent Development for each cohort. The impact at follow-up was 
calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent Development schools. 

Estimates are regression-adjusted using ordinary least squares, controlling for 7th-grade math and reading SAT-9 test scores, race, and whether the student 
had repeated a prior grade. 




Appendix Table B.l (continued) 



A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I, 
and to the impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

■Attendance rates were calculated for each student by dividing the number of days the student was present by the total number of days the student was 
enrolled in a given school year. 

b Until the 1998-1999 school year, 9th-grade students in the district were required to earn four course credits in order to be promoted. Beginning in the 1998- 
1999 school year, minimum requirements for promotion included earning at least five credits during the 9th grade, with three of those credits awarded for 
completing one required course in mathematics, one in English, and one in science. In recent years, the distributive requirement has been droped. 

u 'Basic academic curriculum” is a designation indicating that a student completed a basic academic curriculum, earning at least five credits during the 9th 
grade, with three of those credits being in mathematics, English, and science. Because this designation relied on the district’s course credit code, which does not 
distinguish between elective courses and required courses, the designation may include credits that students earned for elective courses or for required courses. 
Some elective courses did not meet the district’s new 1998-1999 promotion requirements, but all of them counted toward total credits earned. 




The Talent Development Evaluation 
Appendix Table B.2 

Outcome Levels and Impacts on Outcomes 
Measured at the End of the Third Year of High School 
for Repeating Ninth-Grade Students, Follow-Up Results by Cohort 



I. Outcome Levels Compared with Baseline Average 



Talent Development Schools 


Non-Talent Development Schools 


Year 1 Year 2 Year 3 


Year 1 Year 2 Year 3 



Outcome 


Cohort 


Cohort 


Cohort 




Cohort 


Cohort Cohort 


Enrollment status'* (%) 














Enrolled in the 10th grade 


46.9 


47.9 






46.7 


51.9 


Deviation from baseline 


0.8 


1.8 






-1.1 


4.1 


Enrolled in the 9th grade 


43.6 


44.4 






46.3 


42.2 


Deviation from baseline 


-2.4 


-1.6 






5.3 ** 


1.2 


Exited the school system 


9.5 


7.7 






6.9 


5.8 


Deviation from baseline 


1.5 


-0.3 






-4.2 ** 


_5 j *** 


Course credits earned 














Total credits earned over first two years 


6.8 


7.3 






7.8 


8.0 


Deviation from baseline 


0.2 


0.7 






i o *** 


j 2 *** 


At least 2 credits in English 














and at least 2 credits in math (%) 


20.7 


27.1 






26.4 


30.6 


Deviation from baseline 


2.7 


9.1 ** 






6.9 *** 


no *** 








II. Impact of Talent Development 








Year 1 




Year 2 




Year 3 


Outcome 




Cohort 




Cohort 




Cohort 


Enrollment status'* (%) 














Enrolled in the 10th grade 




1.9 




-2.3 






Enrolled in the 9th grade 




-7.7 




-2.8 






Exited the school system 




5.7 




5.1 






Course credits earned 














Total credits earned over first two years 
At least 2 credits in English 




-0.8 




-0.5 






and at least 2 credits in math (%) 




-4.2 




-1.9 
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Appendix Table B.2 (continued) 



SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent Development high schools. Students in the sample 
were included on the district's transcript and attendance records. The sample excludes students who did not attempt at least one credit during a given school 
year. Repeating 9th-grade students were defined as students whose records indicate that they were in the 9th grade in the year under study and were also in the 
9th grade in the previous year's administrative data file. 

The impacts at follow-up were calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent 
Development schools. Impacts are pooled across five clusters of Talent Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development implementation in their school. (The first calendar year of 
implementation varied by school cluster.) Year 2 Cohort includes students who began 9th grade during the second year of Talent Development implementation 
in their school. Year 3 Cohort includes students who began 9th grade during the third year of Talent Development implementation in their school. 

Panel I: Each outcome has two rows of data. The first row shows the average level for that outcome in each cohort. The second row shows the outcome 
level's average deviation from the three-year pre-implementation baseline average in each cohort. 

Panel II: Each outcome has a single row of data that shows the estimated impact of Talent Development for each cohort. The impact at follow-up was 
calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent Development schools. 

Estimates are regression-adjusted using ordinary least squares, controlling for 7th-grade math and reading SAT-9 test scores, race, and whether the student 
had repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I, 
and to the impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

a For the purposes of this analysis, 9th-grade students were considered to have been promoted to 10th grade if they were listed as lOth-graders in the next 
year's administrative data file. 
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Appendix Table B.3 



Outcome Levels and Impacts on Outcomes 
Measured at the End of the Fourth Year of High School 
for Repeating Ninth-Grade Students, Follow-Up Results by Cohort 









I. Outcome Levels Compared with Baseline Average 






Talent Development Schools 


Non-Talent Development Schools 




Year 1 


Year 2 


Year 3 


Year 1 


Year 2 Year 3 


Outcome 


Cohort 


Cohort 


Cohort 


Cohort 


Cohort Cohort 


Enrollment status ' 1 (%) 












Enrolled in the 10th grade 


23.3 






29.8 




Deviation from baseline 


-6.3 ** 






1.4 




Enrolled in the 10th grade 


44.6 






47.2 




Deviation from baseline 


5.8 






9 4 *** 




Exited the school system 


32.1 






23.1 




Deviation from baseline 


0.4 






-10.9 *** 




Course credits earned 












Total credits earned over first three years 


8.39 






9.67 




Deviation from baseline 


0.07 






j i7 *** 




At least 3 credits in English 












and at least 3 credits in math (%) 


16.9 






20.8 




Deviation from baseline 


0.9 






4 7 *** 




At least 17.5 total credits b (%) 


17.9 






21.0 




Deviation from baseline 


-0.9 






3.3 ** 










II. Impact of Talent Development 




Outcome 


Year 1 Cohort 


Year 2 Cohort 


Year 3 Cohort 


Enrollment status 11 (%) 












Enrolled in the 10th grade 




-7.7 


** 






Enrolled in the 9th grade 




-3.6 








Exited the school system 




11.3 








Course credits earned 












Total credits earned over first three years 




- 1.10 








At least 3 credits in English 












and at least 3 credits in math (%) 




-3.8 








At least 17.5 total credits 11 (%) 




-4.3 












Appendix Table B.3 (continued) 

SOURCE: MDRC calculations from individual students’ school records from the School District of Philadelphia. 

NOTES: Sample includes 9th-grade students from five Talent Development high schools and six non-Talent Development high schools. Students in the sample 
were included on the district's transcript and attendance records. The sample excludes students who did not attempt at least one credit during a given school 
year. Repeating 9th-grade students were defined as students whose records indicate that they were in the 9th grade in the year under study and were also in the 
9th grade in the previous year's administrative data file. 

The impacts at follow-up were calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent 
Development schools. Impacts are pooled across five clusters of Talent Development schools and their matched comparison schools. 

Year 1 Cohort includes students who began 9th grade during the first year of Talent Development implementation in their school. (The first calendar year of 
implementation varied by school cluster.) Year 2 Cohort includes students who began 9th grade during the second year of Talent Development implementation 
in their school. Year 3 Cohort includes students who began 9th grade during the third year of Talent Development implementation in their school. 

Panel I: Each outcome has two rows of data. The first row shows the average level for that outcome in each cohort. The second row shows the outcome 
level's average deviation from the three-year pre-implementation baseline average in each cohort. 

Panel II: Each outcome has a single row of data that shows the estimated impact of Talent Development for each cohort. The impact at follow-up was 
calculated as the difference in deviations from the baseline average between Talent Development schools and non-Talent Development schools. 

Estimates are regression-adjusted using ordinary least squares, controlling for 7th-grade math and reading SAT-9 test scores, race, and whether the student 
had repeated a prior grade. 

A two-tailed t-test was applied to the deviations from baseline for Talent Development schools and non-Talent Development comparison schools in Panel I, 
and to the impacts at follow-up in Panel II. Standard errors and statistical significance levels were adjusted to account for cohort effects. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

‘‘For the purposes of this analysis, 9th-grade students were considered to have been promoted to 1 1th grade if they were listed as 1 lth-graders in the 
administrative data file two years after the current year. 

b Students are required to earn 17.5 credits in order to be promoted to the 12th grade. 
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About MDRC 



MDRC is a nonprofit, nonpartisan education and social policy research organization. We 
are dedicated to learning what works to improve the well-being of low-income people. 
Through our research and the active communication of our findings, we seek to enhance 
the effectiveness of social policies and programs. MDRC was founded in 1974 and is 
located in New York City and Oakland, California. 

MDRC’s current projects focus on welfare and economic security, education, and 
employment and community initiatives. Complementing our evaluations of a wide range 
of welfare refonns are new studies of supports for the working poor and emerging 
analyses of how programs affect children’s development and their families’ well-being. 

In the field of education, we are testing reforms aimed at improving the performance of 
public schools, especially in urban areas. Finally, our community projects are using 
innovative approaches to increase employment in low-income neighborhoods. 

Our projects are a mix of demonstrations — field tests of promising program models — 
and evaluations of government and community initiatives, and we employ a wide range 
of methods to determine a program’s effects, including large-scale studies, surveys, case 
studies, and ethnographies of individuals and families. We share the findings and lessons 
from our work — including best practices for program operators — with a broad 
audience within the policy and practitioner community, as well as the general public and 
the media. 

Over the past quarter century, MDRC has worked in almost every state, all of the nation’s 
largest cities, and Canada. We conduct our projects in partnership with state and local 
governments, the federal government, public school systems, community organizations, 
and numerous private philanthropies. 




